DOCUMENT  RESUME 


ED  436  128 

AUTHOR 

TITLE 


INSTITUTION 

PUB  DATE 
NOTE 

PUB  TYPE 
EDRS  PRICE 
DESCRIPTORS 


ABSTRACT 

include:  challenges  for  emerging  instructional  designers;  instructional 
technology  clinical  experience;  color  coding  and  field  dependence;  effects  of 
visualization  on  cognitive  development;  effects  of  learning  structure  and 
summarization  during  computer-based  instruction;  individually-guided 
education  and  problem-based  learning;  World  Wide  Web-based  menus; 
computer-based  training  design;  instructional  principles  for  self  regulation; 
a systems  approach  new  technology  adoption;  navigational  structure  on  the 
Web;  impact  of  navigational  models  on  task  completion  in  Web-based 
information;  multidimensional  knowledge  structures;  relationships  between 
navigational  patterns  and  informational  processing  styles  of  hypermedia 
users;  integrating  science/mathematics  curricula  using  computer-mediated 
communications;  dual -structure  instructional  systems  approach;  effects  of 
metacognitive  training;  evaluating  narrative  simulation  for  potential  to 
impact  bias;  evaluation  theory  in  problem-based  learning;  equivalency  theory 
and  distance  education;  instructional  design  for  distance  education;  using 
multimedia  to  counter  stereotypes;  accessibility  of  Web-based  instruction  for 
students  with  disabilities;  audio  in  computer-based  instruction;  distributed 
cognition,  activity  theory,  and  cognitive  tools;  learner- teacher  interaction 
and  time  spent  by  teaching  faculty  on  distance  education;  effects  of  faculty 
motivation  in  distributive  higher  education;  impact  of  instructional  grouping 
on  navigation  and  student  learning  in  a Web-based  environment;  CD  tutorials 
and  microworlds;  designing  instruction  for  the  technology- supported  K-12 
classroom;  instruct ionist  versus  constructionist  Web-based  collaborative 
learning  environments;  models  for  Web-based  instruction;  multiple 
instructional  design  models;  strategies  for  learner  involvement;  comparison 
of  generative  learning  strategies;  a decision  model  for  problem  selection  in 
problem-based  learning;  relationship  between  teachers  and  instructional 
designers;  goals,  agency  beliefs,  and  emotions  in  Web-assisted  learning; 
historical  analysis  of  problem-based  learning;  secondary  teachers' 
professional  uses  of  computers;  technology  courses  at  colleges  of  education; 
improving  motivation  in  distance  education;  designing  instruction  for 
critical  thinking  dispositions;  screen  miniatures  in  content-based  software; 
psychological  factors  influencing  Web  navigation;  effects  of  goal  intentions 


IR  019  753 

Sparks,  Kristin  E.,  Ed.;  Simonson,  Michael,  Ed. 

Proceedings  of  Selected  Research  and  Development  Papers 
Presented  at  the  National  Convention  of  the  Association  for 
Educational  Communications  and  Technology  [AECT]  (21st, 
Houston,  Texas,  February  10-14,  1999). 

Association  for  Educational  Communications  and  Technology, 
Washington,  DC. 

1999-02-00 

546p . ; For  selected  individual  papers,  see  IR  019  754-811. 
For  the  1998  proceedings,  see  ED  423  819. 

Collected  Works  - Proceedings  (021) 

MF02/PC22  Plus  Postage. 

♦Computer  Uses  in  Education;  *Distance  Education; 
♦Educational  Technology;  Elementary  Secondary  Education; 
Higher  Education;  Hypermedia;  * Instructional  Design;  Problem 
Based  Learning;  World  Wide  Web 

Subjects  addressed  by  the  65  papers  in  this  proceedings 


Reproductions  supplied  by  EDRS  are  the  best  that  can  be  made 
from  the  original  document. 


on  problem  solving  and  reading  comprehension  in  generative  hypertext 
processing;  project-based  learning  with  the  Web;  maximizing  bandwidth; 
interactive  strategy  for  mental  models  learning;  using  scenarios  in 
instructional  design;  educational  technology  doctoral  dissertation  research, 
1977-98;  using  groupware  to  support  collaborative  activities  in  distance 
learning;  influence  of  designer/contextual  variables  on  incorporation  of 
motivational  components;  the  ARCS  (Attention,  Relevance,  Confidence, 
Satisfaction)  model;  technology,  teaching,  and  learning  in  Iowa  high  schools; 
discourse  analysis  as  a tool  for  understanding  user  navigation  models; 
person- environment  interaction  in  the  virtual  classroom; 
asynchronous /synchronous  performance  solutions;  and  designing  online 
instruction  using  multidisciplinary  approaches.  An  author  and  keyword  index 
is  included.  (MES) 


O 

ERIC 


Reproductions  supplied  by  EDRS  are  the  best  that  can  be  made 
from  the  original  document. 


ED  436  128 


Proceedings  of  Selected  Research  and  Development  Papers  Presented  at  the 
National  Convention  of  the  Association  for  Educational  Communications 

and  Technology  (AECT) 

Sponsored  by  the  Research  and  Theory  Division 
Houston,  Texas 
February  10-14,  1999 


Editors: 

Kristin  E.  Sparks  and  Michael  Simonson 


/ 


00 

in 

r 


PERMISSION  TO  REPRODUCE  AND 
DISSEMINATE  THIS  MATERIAL  HAS 
BEEN  GRANTED  BY 

jr._2_e_oo_Y: 


TO  THE  EDUCATIONAL  RESOURCES 
INFORMATION  CENTER  (ERIC) 


n ,.M- s ■ PA RTM ENT  OF  EDUCATION 

Office  of  Educational  Research  and  Improvement 

EDUCATIONAL  RESOURCES  INFORMATION 
CENTER  (ERIC) 

□ This  document  has  been  reproduced  as 
received  from  the  person  or  organization 
originating  it. 

□ Minor  changes  have  heen  made  to 
improve  reproduction  quality. 


9s 


1 


Points  of  view  or  opinions  stated  in  this 
document  do  not  necessarily  represent 
official  OERI  position  or  policy. 


TABLE  OF  CONTENTS 

COMPELLING  CASE  EXPERIENCES:  CHALLENGES  FOR  EMERGING  INSTRUCTIONAL 
DESIGNERS:  by  Marti  F.  Julian,  Valerie  A.  Larsen  & Mable  B.  Kinzie 7 

THE  INSTRUCTIONAL  TECHNOLOGY  CLINICAL  EXPERIENCE:  EXPECTATIONS  AND 
REALITIES:  By  Katherine  S.  Cennamo  & Glen  Holmes 29 

A REVIEW  OF  COLOR  CODING  AND  FIELD  DEPENDENCE  RESEARCH:  By  Francis  M.  Dwyer  & David 
M.  Moore ™ 

EFFECT  OF  VISUALIZATION  (JOB  AIDS)  IN  FACILITATING  COGNITIVE  DEVELOPMENT:  By 
Karen  L.  Spaulding  & Francis  Dwyer ^ 

EFFECTS  OF  LEARNING  STRUCTURE  AND  SUMMARIZATION  DURING  COMPUTER-BASED 
INSTRUCTION:  By  Jenny  Lynn  Wemer  & James  D.  Klein 53 


USING  COOPERATIVE  LEARNING  AND  OBJECTIVES  WITH  COMPUTER-BASED  INSTRUCTION: 
By  James  D.  Klein  & Jamie  C.  Cavalier ^3 

INDIVIDUALLY-GUIDED  EDUCATION  AND  PROBLEM-BASED  LEARNING:  A COMPARISON  OF 
PEDAGOGICAL  APPROACHES  FROM  DIFFERENT  EPISTEMOLOGICAL  VIEWS:  By  B.A. 

Bichelmeyer  & Yu-chen  Hsu ' ' 

WEB-BASED  MENUS:  FONT  SIZE  AND  LINE  SPACING  PREFERENCES:  By  Janice  Pacheco,  Barbara 
Taylor  Day,  Susan  Cribelli,  John  Jordan,  Brandon  Murry  & Kay  A.  Persichitte 81 

CBT  DESIGN:  A FEEDBACK  APTITUDE  TREATMENT  INTERACTION:  By  Roy  B.  Clariana 87 

INSTRUCTIONAL  PRINCIPLES  FOR  SELF  REGULATION:  By  Kathryn  Ley  & Dawn  B.  Young 93 

A SYSTEMS  APPROACH  TO  ADOPTING  NEW  TECHNOLOGIES  IN  EDUCATION:  By  Jared  A. 
Danielson  & John  K.  Burton 


NAVIGATIONAL  STRUCTURE  ON  THE  WORLD  WIDE  WEB:  USABILITY  CONCERNS,  USER 
PREFERENCES,  AND  “BROWSING  BEHAVIOR”:  By  Theodore  Frick,  John  A.  Monson,  Richard  F.  Xaver, 
Gulsen  Kilic,  Aaron  T.  Conley  & Beatrice  Wamey m 

IMPACT  OF  NAVIGATIONAL  MODELS  ON  TASK  COMPLETION  IN  WEB-BASED  INFORMATION 
SYSTEMS:  By  Theodore  Frick,  Eric  Kisling,  Weijia  Cai,  Byeong  Min  Yu,  Frank  Giles  & J.  P.  Brown 119 

MULTIDIMENSIONAL  KNOWLEDGE  STRUCTURES:  By  Kathy  L.  Schuh 127 

THE  RELATIONSHIPS  BETWEEN  NAVIGATIONAL  PATTERNS  AND  INFORMATIONAL 
PROCESSING  STYLES  OF  HYPERMEDIA  USERS:  By  Mi  Jar  Lee  & Francis  A.  Harvey 137 


INTEGRATING  SCIENCE  AND  MATHEMATICS  CURRICULA  USING  COMPUTER  MEDIATED 
COMMUNICATIONS:  A VYGOTSKIAN  PERSPECTIVE:  By  Christina  Wotell  Chamitski  & Francis  A.. 
Harvey — 

DUAL  STRUCTURED  INSTRUCTIONAL  SYSTEMS  APPROACH:  AN  INTEGRATION  OF  SYSTEMIC 
AND  SYSTEMATIC  APPROACHES  FOR  INSTRUCTIONAL  DESIGN:  By  Su-Hong  Park,  Young  Hwan  Kim 
& Yung  Kye  Byun - ^ 


THE  EFFECTS  OF  META  COGNITIVE  TRAINING  ON  PERFORMANCE  AND  USE  OF 
METACOGNITIVE  SKILLS  IN  SELF-DIRECTED  LEARNING  SITUATIONS:  By  Joyce  Kincannon,  Conrad 

171 

Gleber  & Jaehyun  Kim . * 1 


3 


2 


EVALUATING  NARRATIVE  SIMULATION  AS  INSTRUCTIONAL  DESIGN  FOR  POTENTIAL  TO 


IMPACT  BIAS  AND  DISCRIMINATION:  By  Nancye  McCrary  & Joan  M.  Mazur 187 

EVALUATION  THEORY  IN  PROBLEM-BASED  LEARNING  APPROACH:  By  Yu-chen  Hsu 199 

EQUIVALENCY  THEORY  AND  DISTANCE  EDUCATION:  By  Michael  Simonson 207 

BEST  OF  BOTH  WORLDS:THE  NOVA  ITDE  MODEL  OF  DISTANCE  EDUCATION:  By  Charles  A. 
Schlosser  & Marsha  Burmeister 211 


INSTRUCTIONAL  DESIGN  FOR  DISTANCE  EDUCATION:  By  Sharon  Smaldino  & Michael  Simonson...  215 


USING  MULTIMEDIA  TO  COUNTER  STEREOTYPES  IN  SCIENCE  CLASSROOMS:  NEW 
PERCEPTIONS  OF  WHO  BECOMES  A SCIENTIST:  By  Leslie  D.  Edwards 221 

EVALUATING  THE  ACCESSIBILITY  OF  WEB-BASED  INSTRUCTION  FOR  STUDENTS  WITH 
DISABILITIES:  By  D.  Michelle  Hinn 233 

THE  USE  OF  AUDIO  IN  COMPUTER-BASED  INSTRUCTION:  By  Carol  M.  Koroghlanian  & Howard  J. 
Sullivan 241 

WHERE  IS  MY  BRAIN?:  DISTRIBUTED  COGNITION,  ACTIVITY  THEORY,  AND  COGNITIVE 
TOOLS:  By  Linda  S.  Gilbert 249 

DISTANCE  EDUCATION:  LEARNER-TEACHER  INTERACTION  AND  TIME  SPENT  BY  TEACHING 
FACULTY:  By  Veena  Mahesh  & Marina  Stock  Mclsaac 259 

EFFECTS  OF  FACULTY  MOTIVATION  IN  DISTRIBUTIVE  EDUCATION  ENVIRONMENTS  AT 
INSTITUTIONS  OF  HIGHER  EDUCATION:  By  Marlon  R.  Mitchell 269 

IMPACT  OF  INSTRUCTIONAL  GROUPING  ON  NAVIGATION  AND  STUDENT  LEARNING  IN  A 
WEB-BASED  LEARNING  ENVIRONMENT:  By  Angel  Jannasch-Pennell,  Samuel  A.  DiGangi,  Alex  Yu.  Sandra 
Andrews  & John  S.  Babb 275 

WHICH  WAY  TO  JUMP:  CONVENTIONAL  FROG  DISSECTION,  CD-TUTORIAL,  OR 
MICROWORLD?:  By  Christine  S.  Marszalek  & James  Lockard 281 


DESIGNING  INSTRUCTION  FOR  THE  TECHNOLOGY-SUPPORTED  K-12  CLASSROOM:  By  Greg 
Sherman 

INSTRUCTIONS  VERSUS  CONSTRUCTIONIST  WEB-BASED  COLLABORATIVE  LEARNING 


ENVIRONMENTS:  By  Greg  Sherman 311 

MODELS  FOR  WEB-BASED  INSTRUCTION:  A DISCUSSION  OF  RECURRING  THEMES:  By  Craig 
Edwards 319 

MULTIPLE  INSTRUCTIONAL  DESIGN  MODELS:  NECESSITY  OR  EXCESS  BAGGAGE?:  By  David 
Beck 323 

WHAT  CAN  WE  LEARN  FROM  COMPUTER  GAMES:  STRATEGIES  FOR  LEARNER 
INVOLVEMENT:  By  Marshall  G.  Jones 327 

COMPARISON  OF  GENERATIVE  LEARNING  STRATEGIES:  By  Christian  Volk  & Donn  Ritchie 333 

PROBLEM-BASED  LEARNING:  A DECISION  MODEL  FOR  PROBLEM  SELECTION:  By  Jiunde  Lee  341 

TEACHING  INSTRUCTIONAL  DESIGN:  REFRAMING  THE  RELATIONSHIPS  BETWEEN  TEACHERS 
AND  DESIGNERS:  By  Susan  G.  Magliaro  & R.  Neal  Shambaugh 349 


4 


3 


MOTIVATION:  GOALS,  AGENCY  BELIEFS,  AND  EMOTIONS  IN  WEB  ASSISTED  LEARNING;  By  Yi^ 
Guan,  Jianjun  Wang,  Michael  Young,  Steve  Owen  & Richard  Andrew 3a9 

PROBLEM  BASED  LEARNING:  A HISTORICAL  ANALYSIS:  By  Alan  Januszewski  & Robert  Pearson....  371 

SECONDARY  TEACHERS’  PROFESSIONAL  USES  OF  COMPUTERS:  By  Robin  T.  Chiero 377 

RESEARCH  ON  TECHNOLOGY  COURSES  AT  COLLEGES  OF  EDUCATION:  By  Amy  Sheng  Chieh  Leh 


PORTRAYING  THE  WORK  OF  INSTRUCTIONAL  DESIGNING:  AN  ACTIVITY -ORIENTED 
ANALYSIS:  By  Gail  A.  Rathbun 391 

DEVELOPMENT  RESEARCH  APPLIED  TO  IMPROVE  MOTIVATION  IN  DISTANCE  EDUCATION: 

By  Lya  Visser,  Tjeerd  Plomp  & Wilmad  Kuiper 403 

FROM  ABILITY  TO  ACTION:  DESIGNING  INSTRUCTION  FOR  CRITICAL  THINKING 
DISPOSITIONS:  By  Lars  F.  Leader  & James  A.  Middleton 413 

PLANNING  AND  IMPLEMENTING  WEB-BASED  INSTRUCTION:  TOOLS  FOR  DECISION  ANALYSIS: 
By  Stephen  W.  Harmon  & Marshall  G.  Jones 433 

SCREEN  MINIATURES  AS  ICONS  FOR  BACKWARD  NAVIGATION  IN  CONTENT-BASED 
SOFTWARE:  By  Elizabeth  Boling,  Guoping  Ma,  Chia-Wen  Tao,  Cengiz  Askun,  Tim  Green,  Theodore  Frick  & 

429 

Heike  Schaumburg 

PSYCHOLOGICAL  FACTORS  INFLUENCING  WORLD-WIDE  WEB  NAVIGATION:  By  Amy  L.  Baylor 

437 


THE  EFFECTS  OF  GOAL  INTENTIONS  ON  PROBLEM  SOLVING  AND  READING  COMPREHENSION 
IN  GENERATIVE  HYPERTEXT  PROCESSING:  By  Jianjun  Wang,  Michael  F.  Young,  Sasha  A.  Barab  & Yi 


PROJECT-BASED  LEARNING  WITH  THE  WORLD  WIDE  WEB:  A QUALITATIVE  STUDY  OF 
RESOURCE  INTEGRATION:  By  Susan  M.  Land  & Barbara  Greene 451 

MAXIMIZING  BANDWIDTH:  GUIDANCE  FOR  THE  DESIGN  OF  NETWORKED  INSTRUCTION:  By 
David  R.  Moore,  Barbara  B.  Lockee  & D.  Michael  Moore 461 


NO  COMPARISON:  DISTANCE  EDUCATION  FINDS  A NEW  USE  FOR  “NO  SIGNIFICANT 
DIFFERENCE”:  By  Barbara  B.  Lockee,  John  K.  Burton  & Lawrence  H.  Cross 

STUDENT  MODEL  CONSTRUCTION  :AN  INTERACTIVE  STRATEGY  FOR  MENTAL  MODELS 
LEARNING:  By  Ray  Bland  & Martin  Tessmer 

PINNING  THE  CORNERS:  USING  SCENARIOS  IN  INSTRUCTIONAL  DESIGN:  By  Jason  Cole  ...... 


THE  MAJOR  THEMES  AND  TRENDS  IN  DOCTORAL  DISSERTATION  RESEARCH  IN 
EDUCATIONAL  TECHNOLOGY  FROM  1977  THROUGH  1998:  By  Edward  P.  Caffarella ....  483 

A CASE  STUDY  OF  USING  GROUPWARE  TO  SUPPORT  COLLABORATIVE  ACTIVITIES  IN  A 
DISTANCE  LEARNING  COURSE:  By  Lara  M.  Luetkehans 491 

THE  INFLUENCE  OF  DESIGNER  AND  CONTEXTUAL  VARIABLES  ON  THE  INCORPORATION  OF 
MOTIVATIONAL  COMPONENTS  TO  INSTRUCTIONAL  DESIGN  AND  THE  PERCEIVED  SUCCESS 
OF  A PROJECT:  By  Bonnie  J.  Shellnut 503 


THE  ARCS  MODEL  FOR  DEVELOPING  MOTIVATIONALLY -ADAPTIVE  COMPUTER-ASSISTED 


INSTRUCTION:  by  Sang  H.  Song  & John  M.  Keller 513 

INTERNATIONAL  TEAM  TEACHING:  A PARTNERSHIP  BETWEEN  MEXICO  AND  TEXAS:  By  Lauren 
Cifuentes  & Karen  L.  Murphy 531 

TECHNOLOGY,  TEACHING,  AND  LEARNING  IN  IOWA  HIGH  SCHOOLS:  By  Nancy  Maushak,  Lynn 
Mantemach  Wigans  & Caryl  Bender 537 

USING  DISCOURSE  ANALYSIS  AS  A RESEARCH  TOOL  FOR  UNDERSTANDING  USER 
NAVIGATION  MODELS:  By  Richard  J.  Voithofer 543 

PERSON-ENVIRONMENT  INTERACTION  IN  THE  VIRTUAL  CLASSROOM:  AN  INITIAL 
EXAMINATION:  By  Susan  M.  Powers,  Michaeleen  Davis  & Eileen  Torrence 553 

ASYNCHRONOUS  AND  SYNCHRONOUS  PERFORMANCE  SOLUTIONS:  DEVELOPING  A 
COMMUNITY  OF  TECHNOLOGY-USING  TEACHERS:  By  Annette  C.  Sherry,  E.  Barbara  Klemm,  R,  Ian 
McKay,  Jim  Edens  & Patty  Miller 561 

DESIGNING  ON-LINE  INSTRUCTION  USING  MULTIDISCIPLINARY  APPROACHES:  By  Sandra 
Andrews,  Samuel  A.  DiGangi,  David  Winograd,  Angel  Jannasch-Pennell  & Chong-Ho  Yu 565 


AUTHOR  & KEYWORD  INDEX 


573 


d 


Annual  Proceedings 

Of 


Selected  Research  and  Development  Papers 
Presented  at 

The  National  Convention  of  the  Association  for  Educational  Communications  and  Technology 
Sponsored  by  the  Research  and  Theory  Division 
Houston,  TX 
1999 


Editors 

Kristin  E.  Sparks 
RTS  & Associates,  Inc 
Columbus,  OH 
And 

Michael  Simonson 
Professor 

Instructional  Technology  and  Distance  Education 
Nova  Southeastern  University 

Fischler  Graduate  School  of  Education  and  Human  Services 
North  Miami  Beach,  FL 


EKLC 


7 


Previous  Proceedings  Published  in  ERIC 


Year 

Location 

ED  Number 

1979 

New  Orleans 

171329 

1980 

Denver 

194061 

1981 

Philadelphia 

207487 

1982 

Dallas 

223191  - 223326 

1983 

New  Orleans 

231337 

1984 

Dallas 

243411 

1985 

Anaheim 

256301 

1986 

Las  Vegas 

267753 

1987 

Atlanta 

285518 

1988 

New  Orleans 

295621 

1989 

Dallas 

308805 

1990 

Anaheim 

323912 

1991 

Orlando 

334969 

1992 

Washington,  D.C. 

347970-348041 

1993 

New  Orleans 

362144 

1994 

Nashville 

373774 

1995 

Anaheim 

383284 

1996 

Indianapolis 

397772 

1997 

Albuquerque 

409832 

1998 

St.  Louis 

423819 

11 


8 


Preface 


For  the  twenty-first  year,  the  Research  and  Theory  Division  of  the  Association  for  Educational  Communications  and 
Technology  (AECT)  is  sponsoring  the  publication  of  these  Proceedings.  Papers  published  in  this  volume  were 
presented  at  the  National  AECT  Convention  in  Houston,  TX.  A limited  quantity  of  these  Proceedings  were  printed 
and  sold.  It  is  also  available  on  microfiche  through  the  Educational  Resources  Clearinghouse  (ERIC)  system. 

REFEREEING  PROCESS:  Papers  selected  for  presentation  at  the  AECT  Convention  and  included  in  these 
Proceedings  were  subjected  to  a rigorous  blind  reviewing  process.  All  references  to  authorship  were  removed  from 
proposals  before  they  were  submitted  to  referees  for  review.  Approximately  fifty  percent  of  the  manuscripts 
submitted  for  consideration  were  selected  for  presentation  at  the  convention  and  for  publication  in  these  Proceedings. 
The  papers  contained  in  this  document  represent  some  of  the  most  current  thinking  in  educational  communications 
and  technology. 

This  volume  is  indexed  by  both  author  and  descriptor.  The  index  for  volumes  1 -6  (1979  - 1984)  is  included  in  the 
1986  Proceedings,  and  the  index  for  volumes  7 - 10  is  in  the  1988  Proceedings.  After  1988,  each  volume  contains 
indexes  for  that  year  only. 
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Do  Yourself  a Favor . . . 
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division  promotes  the  networking  ol  its  members  to  facilitate  sharing  ol 
expertise  and  interests. 

Division  of  Learning  and  Performance  Environments  (DLPE) 
supports  human  learning  and  performance  through  the  use  ol  computer- 
based  technology,  design,  development,  evaluation,  assessment,  and 
implementation  ol  learning  environments  and  performance  support 
systems  lor  adults. 

Division  of  Telecommunications  (DOT)  strives  to  improve  instruction 
through  the  use  of  telecommunications  including  television,  radio,  video, 
audio,  recordings,  teleconferences,  and  interactrve  distance  education. 


DOT  is  concerned  wilh  the  design,  production,  delivery,  utilization,  and 
evaluation  ol  instructional  telecommunications  materials  and  equipment. 

Division  of  School  Media  Specialists  (DSMS)  enhances  communica- 
tions among  school  media  personnel  and  is  concerned  wilh  the 
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Compelling  Case  Experiences:  Challenges  for  Emerging  Instructional  Designers 


Marti  F.  Julian 
Valerie  A.  Larsen 
Mable  B.  Kinzie 
University  of  Virginia 


Abstract 


Instructional  design  (ID)  case  studies , which  pose  authentic  ill-defined  design  problems  in  realistic 
environments , can  help  instructional  design  students  bridge  the  gap  between  novice  and  expert  practice. 
Over  the  past  three  years , we  have  explored  aspects  of  this  educational  approach  through  the  development 
of  Web-based  instructional  design  cases  at  the  Curry  School  of  Education,  University  of  Virginia.  This 
paper  begins  with  a look  at  case  study  methods  and  how  they  are  used  to  help  students  gain  experience  in 
ID  practice.  We  describe  the  design  and  development  of  our  Web-based  ID  Case  Competition , the  1998  IT 
Case  Event,  and  follow  with  a report  on  the  competition  involving  seven  universities.  Officials  and 
students  felt  that  the  Case  experience  was  valuable  for  developing  ID  expertise  and  preparing  students  for 
professional  practice.  These  advantages ; and  the  opportunity  for  team  collaboration,  were  noted  as 
motivators  for  participation.  Most  of  the  students  were  enthusiastic  about  the  inclusion  of  emergent  ID 
issues  in  the  case  and  felt  the  experience  expanded  their  knowledge  of  ID  practice  and  application.  We 
close  with  a discussion  of  implications  for  the  preparation  of  instructional  designers  and  recommendations 
for  future  development  activities. 

Challenges  for  Emerging  Instructional  Designers 

The  professional  practice  of  instructional  design  (ID)  requires  high  level  problem-solving,  critical  thinking, 
and  interpersonal  skills;  design  problems  are  complex  and  multi-dimensional.  An  effective  designer  must  be  capable 
of  identifying  critical  issues  in  a situation,  and  more  importantly,  select  those  issue(s)  that  can  or  should  be  addressed 
(Ertmer  & Russell,  1995).  In  a comparative  study  of  novice  and  expert  designers,  Rowland  (1992)  observed  that 
experts  tend  to  interpret  an  instructional  design  problem  as  ill-defined  and  immediately  link  the  given  information  to 
previous  experiences  addressing  similar  problems.  In  this  way,  the  expert  is  able  to  infer  additional  information  in 
order  to  develop  a preliminary  concept  of  what  the  problem  is  and  then  apply  templates  for  examining  the  situation 
such  as  the  needs  analysis  model.  As  a result,  experts  reflect  on  a wide  range  of  factors  and  spend  considerable  time 
interpreting  the  design  problem  (Perez  & Emery,  1995;  Rowland,  1992).  Novices,  on  the  other  hand,  do  not  possess 
this  body  of  knowledge  and,  therefore,  accept  the  given  information  and  move  on  to  generation  of  solutions  without 
developing  an  in  depth  understanding  of  the  problem  itself. 

Instructional  design  is  commonly  taught  as  a set  of  procedures  with  clear  cut  examples  that  do  not  match  the 
complexities  of  real-world  situations  (Rowland,  1992).  Ideally,  students  are  given  the  opportunity  to  apply  what  they 
learn  to  an  actual  design  project  so  they  can  discover  the  strengths  and  limitations  of  the  process  and  theory.  These 
projects  require  considerable  time,  a finite  commodity  in  IT  programs.  Students,  therefore,  are  exposed  to  a limited 
number  of  professional  practice  settings  (Kinzie,  Hrabe,  & Larsen,  1998).  Consequently,  novice  designers  frequently 
enter  the  workforce  with  an  understanding  of  the  ID  process  but  without  the  knowledge  base  that  can  help  them  solve 
instructional  design  problems  and  develop  solutions. 

Case  studies,  which  pose  authentic  ill-defined  design  problems  can  help  bridge  this  gap.  In  case  analysis, 
instructional  design  students  draw  connections  between  their  emerging  knowledge  of  ID  and  the  complex  demands 
of  actual  practice.  Cases  can  supplement  student  design  projects,  allowing  further  opportunity  to  reflect  on  relevant 
theory  and  methods  as  students  explore  a greater  number  of  design  issues  in  a broader  array  of  environments. 
Furthermore,  students  are  provided  an  opportunity  to  broaden  their  knowledge  base  as  they  collaborate  with 
colleagues  to  identify  effective  design  solutions  (Kinzie  et  al.,  1998). 

Over  the  past  three  years,  we  have  explored  aspects  of  this  educational  approach  through  the  development 
of  Web-based  instructional  design  cases  at  the  Curry  School  of  Education,  University  of  Virginia.  The  World  Wide 
Web  enables  students  at  any  institution  or  any  interested  individuals  to  access  our  multimedia  case  studies  and  we 
use  this  venue  to  sponsor  an  annual  competition,  the  IT  Case  Event.  (http://cuiTv.edschool.virginia.edu/go/Itcases) 
Each  year.  Teams  from  up  to  seven  participating  universities  analyze  an  instructional  design  case,  while  expert 
officials  pose  probing  questions,  evaluate  case  responses,  and  contribute  their  own  perspectives  on  the  case. 

Evaluations  of  our  first  two  Case  Events  suggest  that  case  analysis  can  serve  as  a valuable  supplement  and 
may  help  expand  the  depth  and  breadth  of  novice  designers’  expertise  (Kinzie  et  al.,  1998).  In  this  paper,  we  built  on 
this  past  research,  with  the  intent  of  describing  the  case  event  experiences  of  students  and  officials  during  the  1998 
event.  We  identify  skills  used  during  the  case  analysis,  and  offer  some  observations  on  perceptions  of  preparation 
and  challenge.  We  also  examine  the  perceived  value  of  integrating  an  emerging  ID  practice:  the  design  of 
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performance  support.  Our  primary  research  question  was,  "Is  an  instructional  design  case  competition  a worthwhile 
medium  for  expanding  professional  knowledge  and  exploring  emerging  issues  in  instructional  design?"  We  were  also 
interested  in  whether  developments  in  our  case  study  and  approach  to  interface  design  would  enhance  the  sense  of 
realism,  thereby  encouraging  a deeper  exploration  into  the  ID  issues. 

We  begin  this  paper  by  providing  background  information  on  case  study  methods  and  how  they  are  used  to 
help  students  gain  experience  in  instructional  design  practice.  We  follow  with  a description  of  the  1998  IT  Case 
Competition,  focusing  on  the  developments  in  case  design  and  methods  we  implemented.  Evaluative  data  based  on 
the  results  of  the  competition,  follow-up  surveys  and  individual  interviews  serve  as  the  focal  point  of  this  study.  We 
close  with  a discussion  of  implications  for  the  preparation  of  instructional  designers  and  recommendations  for  future 
development  activities. 

Case  Methods  and  Instructional  Design 

Case  methods  have  long  been  used  extensively  in  professions  such  as  law,  business,  and  medicine  and  more 
recently  have  gained  popularity  in  other  professions  such  as  teacher  education,  engineering,  nursing,  and 
instructional  design  (Ertmer  & Russell,  1995).  The  growing  interest  in  this  method  of  instruction  reflects  recent 
advances  in  the  cognitive  sciences,  which  suggest  that  knowledge  and  skills  are  best  learned  in  contexts  that  reflect 
the  way  they  will  be  useful  in  real  life.  Because  of  this  situated  cognition,  the  individual  is  able  to  apply  this 
knowledge  to  new  situations  through  recall  of  the  experience  and  the  environment  in  which  it  occurred  (Bednar, 
Cunningham,  Duffy,  & Perry,  1991;  Teslow,  Carleson,  & Miller,  1994).  Constructivism  builds  on  this  theory  by 
acknowledging  a belief  in  multiple  realities;  each  learner  creates  his  or  her  own  perception  of  reality  based  on 
previous  knowledge  and  experience.  Instruction  that  is  based  on  this  paradigm,  such  as  case  methods,  focuses  on 
student-centered,  active  learning  experiences  where  learners  are  encouraged  to  use  prior  knowledge  to  explore  and 
create  new  interpretations  of  the  world  around  them.  (Perkins,  1992;  Teslow  et  al.,  1994). 

As  the  use  of  cases  has  grown,  case  methods  have  continued  to  evolve,  and  can  now  be  found  in  a variety  of 
forms.  In  teacher-education  and  instructional  design,  case  methods  have  taken  the  form  of  complex  fictional 
narratives  which  are  grounded  in  actual  problems  and  challenges  that  occur  in  the  real  world  (Ertmer  & Russell, 
1995\;  Kinzie  et  al.,  1998).  Grabinger  (1996)  has  identified  these  methods  as  one  way  of  providing  rich 
environments  for  active  learning.  They  allow  students  to  construct  knowledge  in  an  authentic  environment  where 
they  assume  personal  responsibility  for  learning  and  work  cooperatively  to  produce  something  of  real  value. 

Effective  cases  are  those  that  are  immediately  realistic  and  meaningful  to  the  learners  and,  like  professional  practice, 
are  ambiguous  and  messy,  offering  opportunity  for  analysis  and  consideration  of  multiple  solutions.  Presentation 
within  specific  contexts  makes  the  case  more  believable  and  makes  what  is  learned  more  readily  retrieved  from 
memory  when  the  learner  is  placed  in  similar  circumstances  (L.  Shulman  (1992)  as  cited  in  Ertmer,  1995; 

Grabinger,  1996). 

The  cases  we  have  developed  for  the  IT  Case  Event  are  intended  to  serve  as  a useful  supplement  for 
instructional  design  students  by  exposing  them  to  various  professional  practice  settings  in  a context  that  offers 
"enough  depth  and  complexity  to  provide  realistic  challenges"  (Kinzie  et  al.,  1998  p.55).  Since  cases  can  be 
analyzed  in  much  less  time  than  a design  project,  there  is  often  the  opportunity  for  students  to  analyze  multiple  cases. 
As  a result,  students  can  gain  exposure  to  ID  practice  in  industry,  education,  museums,  and  the  military  and  the 
unique  problems  that  may  exist  in  each  type  of  environment  (Kinzie  et  al.,  1998).  We  recommend  that  students 
follow  a collaborative  problem  solving  process  based  on  the  work  of  McNergney,  Herbert,  and  Ford  (1994): 

• identify  key  issues, 

• consider  multiple  perspectives  on  the  events  in  the  narrative, 

• apply  current  professional  knowledge, 

• develop  appropriate  solutions,  and 

• predict  the  consequences  of  their  proposed  courses  of  action. 

The  need  to  solve  problems  through  collaboration,  frequently  with  others  who  have  different  skill  sets  and 
perspectives,  is  a common  occurrence  for  most  instructional  designers.  Students  have  limited  opportunities  to  learn 
how  to  effectively  deal  with  the  conflicting  values  and  multiple  points  of  view  that  are  inherent  in  the  group  work. 
Likewise,  they  have  few  occasions  to  advance  and  develop  support  for  their  own  perspectives.  Case  scenarios 
encourage  professional  collaboration,  providing  students  opportunities  to  improve  their  oral  communication  and 
interpersonal  skills  as  they  discuss,  debate,  justify,  and  integrate  their  ideas  into  what  they  agree  is  the  best  possible 
solution  (Ertmer  & Russell,  1995;  Kent,  Herbert,  & McNergney,  1995;  Stepien  & Gallagher,  1993).  Ellsworth 
(1994)  found  that  collaborating  students  take  on  a more  active  role  in  the  learning  process,  becoming  problem 
solvers,  contributors,  and  discussants.  Competition  can  be  a useful  adjunct  to  collaboration,  allowing  this  experience 
to  reflect  the  real  world,  where  design  Teams  must  often  compete  with  others  to  identify  the  best  possible  solution 
(Kinzie  et  al.,  1998). 
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IT  Case  Event:  Web-based  Case  Media 


Our  Web-based  cases  are  designed  to  emulate  the  complexities  of  real-world  events  within  a selected  area 
of  professional  practice  in  ID.  The  design  and  development  of  each  case  is  a team  effort,  with  selection  of  the  case 
environment  and  characters  based  on  the  previous  experience  of  the  primary  author.  The  design  dilemmas  emerge 
from  this  experience,  the  ID  experience  of  the  team  members,  and  from  problems  alumni  have  encountered  in 
professional  practice  in  particular,  those  that  are  not  addressed  by  standard  ID  theory  and  methods  alone.  We  design 
cases  to  enhance  the  realism  necessary  to  compel  exploration  of  the  ID  issues.  Starting  in  1998,  we  began 
conducting  needs  assessments  in  the  selected  case  environment.  We  conduct  multiple  interviews  of  stakeholders  at 
various  levels  across  several  organizations.  Our  aim  is  to  obtain  stakeholder  perspectives  about  the  environment  and 
the  ID  issues  and  to  identify  typical  characteristics  of  individuals  working  at  various  levels  within  this  industry. 

Next,  the  primary  author  begins  construction  of  the  case  with  an  outline  of  potential  events  and  critical 
issues.  The  team  uses  this  outline  to  brainstorm  potential  case  materials  and  possible  thematic  formats.  The  author 
then  creates  a draft  of  the  entire  case,  attempting  to  build  a realistic  environment  and  believable  interplay  between 
the  characters  and  the  issues  presented. 

Production  of  the  case  materials  involves  coding  the  files  that  present  the  case  documents  in  HTML 
(HyperText  Markup  language),  recruiting  talent  for  the  character  representations,  and  creating  the  media  for  the  case 
(graphics  and  audio  and  video  clips  to  illustrate  the  characters  and  events).  Once  production  begins,  we  start 
prototyping  the  case  with  one-to-one  and  small  group  evaluations,  followed  by  revisions.  Final  modifications  and 
revisions  are  made  once  the  case  has  been  released  and  used  by  a large  group  of  students.  For  the  1998  case,  we  also 
solicited  case  reviews  by  several  of  the  industry  professionals  we  had  interviewed  in  the  needs  assessment  to  check 
for  consistency  and  accuracy  in  the  content. 

We  have  found  that  this  combination  of  text  and  media  on  the  Web  can  foster  both  inductive  and  deductive 
strategies  as  the  students  navigate  through  the  hyperlinked  ideas,  events,  and  artifacts.  For  instance,  students  can 
analyze  ID  methods  that  have  been  integrated  into  the  case  study  or  test  ID  models  and  processes  as  they  map  out 
what  is  known  or  unknown  about  a given  design  problem  or  solution.  This  process  of  reasoning  is  in  line  with  the 
analytical  processes  instructional  designers  must  engage  in  as  they  create  order  and  coherence  out  of  disparate 
sources  of  information  (Kovalchick,  Hrabe,  Kinzie,  & Julian,  1998). 

In  this  study,  we  explore  student  and  official  perceptions  of  reasoning  employed  by  Student  Teams  in  the 
1998  IT  Case  Event.  We  also  consider  whether  the  changes  imparted  in  our  design  and  of  the  Web-based  cases 
enhance  the  learning  experience  of  the  learning  participants.  We  ask  the  following  primary  question: 

• is  a Web-based  case  competition  a worthwhile  medium  for  expanding  professional  knowledge  and 
exploring  emerging  issues  in  instructional  design? 

And  the  following  secondary  questions: 

• Did  this  case  experience  alter  student  perceptions  of  skills  they  need  for  professional  practice  in  ID? 

• Have  the  developments  in  the  design  and  delivery  of  our  Web-based  cases  enhanced  the  sense  of  realism  so 
they  are  compelling  and  sufficiently  complex  for  exploration  into  ID  issues  and  practice? 


Methods 

Participants 

A total  of  42  students,  28  female  and  14  male  participated  in  the  1998  IT  Case  Event.  The  seven  Student 
Teams,  ranging  in  size  from  four  to  seven  members,  were  from  the  following  academic  universities:  Colorado  at 
Colorado  Springs,  Colorado  at  Denver,  Northern  Illinois,  Purdue,  San  Diego  State,  Virginia  Polytechnic  Institute, 
and  Virginia.  The  students  were  enrolled  in  both  master's  and  doctoral  programs  and  six  of  the  Teams  had  received 
training  in  instructional  design  as  part  of  their  programs.  The  seventh  team  was  enrolled  in  an  introductory  design 
course  that  had  met  only  twice  before  the  beginning  of  the  competition  and  its  members  were  participating  for  course 
credit.  Four  of  the  Teams  received  course  credit  for  their  participation  and  three  Teams  participated  as  an  extra- 
curricular activity.  As  in  previous  Case  Events,  the  majority  of  the  students  (72%)  had  more  than  four  years  of  full- 
time work  experience  in  a variety  of  professions  from  K-I2  and  adult  education  to  engineering,  the  military,  and 
business. 

The  seventeen  officials  included  Team  Sponsors,  Provocateurs  and  Judges.  The  Team  Sponsors  nominated 
Student  Teams  and  relayed  case  event  communications  to  the  students.  They  were  also  encouraged  to  analyze  a case 
with  their  Teams  prior  to  the  event.  The'Team  Sponsors  each  nominated  one  or  more  professionals  to  participate  as 
Judges  or  Provocateurs.  Additional  officials  volunteered  or  were  recruited  by  the  development  team. 


Materials 

The  case  developed  for  the  1998  IT  Case  Event,  " The  Chronicles  of  Rocket  Boy takes  place  in  the  digital 
animation  sector  of  the  feature  film  industry  (Julian,  Kinzie,  & Larsen,  1998).  This  fictional  narrative  is  based  on 
real  issues  advanced  by  the  case  authors,  industry  experiences  of  the  authors,  and  the  perspectives  shared  by  the 
individuals  from  the  digital  animation  industry  who  were  interviewed.  This  case  also  advances  ID  problems  selected 
in  advance  by  the  case  authors,  including  the  emerging  practice  of  performance  support.  The  case  is  written  in  the 
first  person,  through  the  eyes  of  an  instructional  designer,  Jason,  (Figure  1)  who  is  brought  into  a large  digital 
animation  house  to  assist  in  designing  an  electronic  performance-support  system  (EPSS).  With  five  years 
instructional  design  experience,  Jason  is  charged  with  the  task  of  identifying  performance-support  measures  to 
increase  productivity  on  the  production  line  and  help  the  company  turn  a profit.  Through  several  interviews,  Jason 
peels  back  the  complex  layers  of  this  organization,  identifying  the  key  issues  that  will  help  him  design  the  best 
possible  solution.  The  case  concludes  as  Jason  begins  to  analyze  the  data  from  his  needs  assessment  and  assures  the 
training  department  team  that  a performance-support  solution  can  be  developed  within  the  nine-month  time  frame 
that  has  been  budgeted  for  this  endeavor. 

Navigation  through  the  site  is  guided  by  a hyperlinked  index  designed  to  look  like  a movie  marquee  listing 
the  events  in  the  case  (Figure  2).  The  user  can  also  navigate  through  hyperlinks  provided  at  the  bottom  each  page. 
Five  case  ancillaries  (letters,  charts,  and  descriptions)  are  presented  as  part  of  an  orientation  packet  for  the 
instructional  designer  (Figure  3).  These  documents  help  depict  the  structure  of  the  company.  Photographs  of 
characters  in  action  and  graphics  dispersed  throughout  the  case  add  depth  to  the  case  events  (Figure  4).  Links  to 
articles  defining  EPSS  and  its  applications  in  ID  were  also  embedded  in  the  case.  The  case  may  be  examined  at  the 
following  URL:  http://curry.edschool.virginia.edu/go/ITcases. 

Figure  1:  Introduction  to  Jason  in  the  Web-based  case,  ”The  Chronicles  of  Rocket Boy" 


Figure  2:  Movie  marquee  as  index  for  the  Web-based  case,  "The  Chronicles  of  RocketBoy 


Figure  3:  Ancillaries  from  the  Web-based  case,  ' The  Chronicles  of  RocketBoy " 
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Figure  4:  Interview  Screens  from  the  Web-based  case,  "The  Chronicles  of  RocketBoy 
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Procedures 

Teams  were  encouraged  to  prepare  for  the  event  by  analyzing  a case  from  previous  events,  and  by 
reviewing  case  responses  and  the  Judges  comments.  Participants  were  also  informed  that  EPSS  would  be  an  issue 
embedded  in  the  case  and  were  encouraged  to  review  the  recommended  articles  that  would  be  linked  into  the  case 
documents.  Once  the  case  for  the  1988  event  was  available,  a three-week  time-period  was  provided  for  analysis 
response  development.  Teams  were  limited  to  meeting  for  a maximum  of  six  hours  with  team  members.  This  time 
limit  applied  to  face-to-face  meetings  and  to  asynchronous  discussions  (such  as  e-mail  communications)  between  as 
few  as  two  team  members  or  with  the  entire  team.  No  limits  were  placed  on  individual  review,  analysis,  or  writing, 
and  Teams  were  encouraged  to  reference  any  resource  materials  they  desired.  Teams  were  instructed,  however,  to 
respond  to  the  case  without  participation  from  others. 

The  entire  case  response  was  limited  to  2000  or  fewer  words  to  encourage  concise  expression  of  ideas  and 
to  keep  the  review  work  of  event  officials  manageable.  In  developing  their  responses,  the  Teams  were  directed  to 
complete  the  needs  assessment,  including  analysis  of  needs  and  alternative  performance  support  solutions.  They  were 
asked  to  include  in  their  final  recommendations  an  outline  of  a preliminary  design  based  on  the  needs  assessment, 
specifying  goals,  strategies,  and  target  groups  where  appropriate  and  to  base  their  case  responses  on  each  of  the 
following  tasks: 


• Identification  of  the  key  issues  present  inthe  case, 

• Consideration  of  the  issues  from  different  perspectives,  including  those  of  the  key  players  (stakeholders)  in  the  case,  and 

• Relevant  professional  knowledge/experience  that  team  members  possess. 

Following  their  review  of  the  submitted  case  responses,  the  four  Provocateurs  met  asynchronously  via  e- 
mail  to  compose  a general  question  for  all  of  the  Teams  and  a specific  question  for  each  team  in  reaction  to  their 
analysis.  The  Teams  were  allowed  up  to  two  hours  working  together  on  their  response  to  the  Provocateur  questions. 
There  was  no  length  limit,  however,  to  this  phase  of  the  case  activity  and  the  Teams  had  one  week  to  produce  a 
response  to  the  questions.  The  case  and  question  responses  were  then  forwarded  to  the  Judges  who  reviewed  the 
materials  individually,  evaluated  the  Teams'  responses  using  a rating  scale,  and  wrote  evaluative  feedback  for  each 
team.  The  ratings  were  tabulated  and  a winner  was  announced.  All  of  the  team  responses  were  posted  along  with  the 
Provocateur  questions  and  Judges'  comments  and  the  participants  were  encouraged  to  review  and  discuss  the  various 
solutions  presented  by  the  Teams.  These  materials  can  be  reviewed  at  the  URL: 
httD://currv.edschool.Virginia.EPU/go/ITcases/Site/98Teams/Outcomes. 


Measures 

While  the  Teams  were  responding  to  the  Provocateur  questions,  the  panel  of  six  Judges  met  via  email  to 
review  the  rating  scale  from  the  previous  Case  Event.  They  adapted  it  from  the  1997  Case  Event  to  reflect  the 
challenges  presented  in  The  Chronicles  of  RocketBoy , adding  three  new  items  pertaining  to  emerging  issues  in  ID 
(see  items  13-15  in  Table  1).  Ratings  were  made  using  a four-point  scale  from  Strongly  Disagree  [1]  to  Strongly 
Agree  [4].  The  following  table  contains  a listing  of  the  response  rating  items.  (Table  1): 
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Table  1:  Judge  Rating  Scale  for  Team  Case  Responses 

1.  The  team  identified  all  of  the  important  issues  in  the  case. 

2.  The  team  effectively  addressed  all  of  the  important  issues  in  the  case. 

3.  The  team  defined  the  perspectives  of  all  the  relevant  stakeholders  in  this  case,  (e.g.,  artists,  technicians,  supervisors, 

administration,  trainers,  organization.) 

4.  The  team  effectively  responded  to  all  of  the  relevant  perspectives  in  this  case. 

5.  The  team  effectively  analyzed  the  needs  identified. 

6.  The  team  identified  appropriate  alternative  solutions  for  each  need. 

7.  The  team  developed  an  instructional  goal  that  was  appropriate  for  the  case. 

8.  The  team  recommended  an  appropriate  overall  solution. 

9.  The  team's  specifications  for  personnel  to  be  involved  in  the  solution  were  appropriate. 

10.  The  team  effectively  integrated  relevant  professional  knowledge  (theories  and  practices)  into  their  response. 

1 1 . Overall,  the  team's  Needs  Assessment  and  Preliminary  Design  were  appropriate  for  the  case. 

12.  Overall,  the  team's  Needs  Assessment  and  Preliminary  Design  demonstrated  excellence. 

13.  The  team  was  proactive  in  making  recommendations  and/or  modifying  the  environment,  as  opposed  to  being  only  reactive  and 
giving  the  client  what  they  said  they  wanted. 

14.  The  team  identified  all  major  project  risks  and  developed  plans  to  manage  them. 

15.  The  team  presented  an  appropriate  quality  management  plan. 


Once  the  students  had  submitted  their  responses  to  the  Provocateur  questions,  we  sent  an  e-rnail  request 
asking  them  to  evaluate  their  Case  Event  experience  by  completing  a Web-based  survey.  We  sent  similar  requests  to 
Provocateurs,  Team  Sponsors,  and  Judges,  as  they  completed  their  contributions  to  the  Case  Event.  Both  students 
and  officials  were  asked  nine  common  questions  with  slight  variations  displayed  in  the  table  below  (Table  2).  In 
addition,  the  students  were  asked  whether  they  had  participated  as  part  of  a class  or  some  other  form  of  academic 
credit.  The  students  were  also  asked  if  there  were  other  factors  that  had  prompted  their  decision  to  participate.  This 
survey  is  closely  aligned  with  the  survey  used  in  the  previous  study,  with  the  addition  of  questions  reflecting  our 
focus  on  emergent  issues  and  our  interest  in  the  application  of  the  case  experience  to  actual  practice.  Responses  to 
the  surveys  were  submitted  electronically  through  the  survey  Web-forms.  They  were  analyzed  using  simple 
descriptive  statistics  (means  and  standard  deviations)  and  simple  qualitative  analysis  for  the  open  ended  questions. 

Table  2:  Survey  Questions  Answered  by  All  Participants  (Text  in  brackets  represent  variation  in  student  Survey) 

Survey  Question Response  Type 

Number  of  years  you  have  worked  in  a field  related  to  ID  [full  time  job]: 

(Officials:  1-5  years,  6-10  years,  11-15  years,  16-20  years,  21  or  more  years)  Multi.  Choice 

(Students:  0 years,  1-5  years,  6-10  years,  11-15  years,  16-or  more  years) 

Prior  to  this  event,  had  you  ever  used  cases  as  a teaching  [learning]  tool?  YES/NO 

If  so  what  was  the  content  (ID,  teacher  education,  law,  medicine)  and  how  did  you  use  the  case(s)?  Please  Fill  in 

describe: 


How  did  you  prepare  for  the  case  competition? 

1 read  previous  ITcases:  Check  boxes 

(Prescription:  Instructional  Design,  Harvesting  Cooperation,  Trials  of  Terry  Kirkland,  None) 

I explored  the  ITcases  Web  site:  Check  boxes 

(Previous  Case  Responses,  Expert  Perspectives  from  Previous  Cases,  Case  Method 
Publications,  None) 

Other  preparation:  Fill  in 

Did  you  have  any  difficulties  accessing  the  case  materials?  YES/NO 

If  so,  please  describe:  Fill  in 

Participation  in  this  case  competition  helped  prepare  the  students  (me)  for  future  instructional  design  Likert  scale 

projects. 

(Strongly  Agree,  Agree,  Disagree,  Strongly  Disagree) 

The  use  of  the  case  study  method  is  valuable  in  developing  expertise  related  to  instructional  design.  Likert  scale 

(Strongly  Agree,  Agree,  Disagree,  Strongly  Disagree) 

In  what  areas  of  professional  development  do  you  feel  the  students  (you)  were  most  prepared  for  the  case  Fill  in 

analysis? 

In  what  areas  were  they  (you)  most  challenged?  Fill  in 

What  issues  presented  in  the  case  do  you  perceive  as  most  relevant  to  actual  practice  in  the  field?  Fill  in 

Taking  this  experience  as  a whole,  what  was  most  valuable?  What  was  least  valuable?  Fill  in 

Do  you  have  any  suggestions  for  future  modifications  of  this  event,  the  case  study,  or  the  Web  site?  Fill  in 
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Participants  who  completed  the  surveys  received  an  email  request  asking  if  they  would  participate  in  a 
telephone  interview.  The  interviews  with  the  participant  volunteers  took  between  25  and  50  minutes,  and  were  tape 
recorded  and  later  transcribed  for  analysis.  The  interview  questions  were  adapted  from  the  previous  study  to  reflect 
our  foci.  A common  set  of  questions  was  posed  to  all  participants  (see  Table  3)  and  specific  questions  were  directed 
to  students  and  officials,  respectively.  Interviewees  were  encouraged  to  elaborate  on  their  ideas  and  add  additional 
comments  at  will  during  the  course  of  the  interview.  Additional  questions  emerged  during  the  interviews  in  response 
to  participant  comments.  Table  4 presents  questions  that  were  addressed  to  students  only,  with  several  questions 
focusing  on  team  process  and  preparation.  Table  5 presents  questions  posed  to  the  officials.  In  addition,  Team 
Sponsors  were  asked  if  they  had  a perspective  on  how  the  Teams  discussed  the  materials.  Provocateurs  were  asked 
how  they  organized  their  approach  to  developing  the  questions,  and  the  Judges  were  asked  how  they  coordinated  the 
evaluation  and  feedback  of  the  case. 

Table  3:  Interview  Questions  Answered  by  All  Participants 

Was  The  Chronicles  of  RocketBoy  case  study  realistic?  Was  it  sufficiently  compelling  that  you  were  able  to  enter  it  and 
explore  the  ID  issues? 

If  so,  what  contributed  to  this  sense? 

If  not,  are  there  any  aspects  of  the  case  that  should  have  been  adapted  to  increase  the  sense  of  realism? 

What  qualities  of  professional  instructional  designers  do  you  feel  the  students  (you)  reflected  during  the  event? 

(Theoretical  base,  management  skills,  organizational  management,  awareness  of  current  trends,  etc...) 

In  what  areas  do  you  feel  the  students  (you)  were  most  prepared  for  the  Case  Event  and  in  what  areas  were  they  (you)  most 
challenged? 

What  issues  presented  in  the  case  do  you  perceive  as  most  relevant  to  actual  practice  in  the  field? 

Table  4:  Interview  Questions  posed  to  Students 

How  did  you  organize  your  team's  approach  to  the  case  analysis? 

How  did  you  assign  responsibilities  among  team  members?  What  role(s)  did  you  have? 

How  did  you  coordinate  the  case  analysis  and  response  generation? 

What  kinds  of  discussions  did  your  team  have?  How  would  you  characterize  them? 

How  did  you  feel  about  them? 

How  did  you  address  conflicting  viewpoints  within  the  team? 

Is  there  any  other  group  dynamic  that  you  think  is  noteworthy? 

Did  knowing  that  the  response  was  going  to  be  judged  influence  your  approach  to  the  activity?  How? 

Did  it  affect  your  performance?  How? 

How  did  you  make  sense  of  the  case? 

(process  information,  organize  it) 

How  did  you  feel  about  your  case  response? 

How  did  this  activity  differ  from  those  you  might  engage  in  within  a class  or  within  your  school’s  program? 

How  did  you  feel  about  the  questions  you  received  from  the  Provocateurs?  Did  they  encourage  your  team  to  consider  other  important 
issues?  In  what  way? 

Do  you  feel  the  case  experience  enhanced  your  skills  as  a designer? 

If  so,  in  what  way? 

If  not,  why? 

Did  you  learn  anything  about  needs  assessment  that  you  didn’t  know  before? 


Table  5:  Interview  questions  posed  to  All  Officials 

Do  you  feel  the  students  had  the  appropriate  skills  to  analyze  the  case? 

Do  you  feel  the  case  analysis  contributes  to  the  development  of  the  students’  professional  skills? 
Do  you  have  a sense  of  what  skills  the  students  used  to  make  sense  of  the  case? 

How  do  you  feel  about  the  coordination  of  the  Case  Event? 


Data  Analysis 

Quantitative  data  were  analyzed  with  simple  descriptive  statistics  (means,  standard  deviations,  percentages). 
The  qualitative  data  included  responses  to  the  open-ended  survey  questions  and  all  of  the  interview  comments.  The 
survey  and  interview  data  from  each  individual  were  labeled  for  cross-referencing  before  creating  categories  to  avoid 
duplication  of  individual  perspectives  in  the  data  analysis.  A conventional  mode  of  qualitative  inquiry  was  employed: 
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In  formulating  the  questions,  the  four  Provocateurs  took  the  role  of  the  company  manager  responding  to  the 
instructional  designer’s  recommendations.  The  common  question  was  posed  to  all  of  the  Teams; 

"Can  we  slow  this  down  a little,  Jason?  You've  only  been  here  for  what,  a month  and  you've  talked  with  a few 
people.  And  based  on  that,  you're  making  an  assortment  of  recommendations  that  resemble  re -constructive  surgery 
on  our  company.  I don’t  think  the  boss  will  be  all  that  impressed  and,  frankly,  it  seems  like  overkill  to  me,  too. 

We're  looking  to  add  an  EPSS,  not  restructure  the  company.  What  elements  of  your  proposal  can  be  implemented, 
realistically,  within  the  existing  corporate  structure  and  timelines  to  get  the  EPSS  going  and  support  its  operation?" 

A specific  question  was  developed  for  each  individual  team  in  reaction  to  the  team's  case  response.  For 
example,  one  team  was  asked  to  evaluate  the  effectiveness  a proposed  action  would  have  in  the  in  the  immediate 
future: 


One  of  your  suggestions  is  to  establish  a debriefing  period  after  each  project  is  completed.  That’s  an  intriguing  idea 
and  I can  see  how  it  might  help  us  identify  successes  and  failures,  but  only  after  the  fact.  What  about  now?  How  do 
you  suggest  we  use  this  idea  to  help  us  develop  our  EPSS? 

Another  team  was  asked  to  consider  the  effects  their  proposed  solution  would  have  on  the  successful 
collaboration  already  in  place  on  the  production  line: 

You've  recommended  a formalized  pipeline-partnering  program  as  part  of  your  OPSS.  There  is  already  a lot  of 
"mentoring"  going  on.  Employees  frequently  consult  with  one  another  and  this  seems  to  be  a very  positive  part  of 
the  pipeline.  My  concern  is  that  a formal  partnering  program  will  be  more  disruptive  than  supportive.  How  do  you 
propose  to  structure  a partnering  program  so  that  it  will  support,  rather  than  disrupt,  the  existing  collaboration? 

The  specific  and  common  questions  and  responses  can  be  viewed  on-line  at:  Case  Event  ’98  ’’Outcomes” 
http://curry.edschool.virginia.edu/tzo/ITcases  ). 

The  case  response  and  response  to  the  Provocateur  questions  were  reviewed  (blind)  by  the  six  Judges  over  a 
two-week  period.  The  quantitative  results  (Table  1)  were  compiled  from  the  Judges’  ratings  of  the  criteria  they  had 
identified  prior  to  the  event  and  followed  up  with  comments  reflecting  on  each  team’s  performance.  Each  team’s 
ratings  were  totaled  and  averaged  across  the  fifteen  items  (Low:i  to  High:4).  The  outcome  of  the  average  team 
ratings  were  fairly  high,  ranging  from  a low  of  2.82  (S£>=0.36)  to  a high  of  3.73  (££>=0.29).  In  general,  the  Judges 
felt  positive  about  the  Teams’  performance  in  their  application  of  ID  theory  and  demonstrated  insight  into 
professional  practice  (see  Table  6).  The  ratings  suggest  the  Teams'  weakest  performance  was  in  their  identification  of 
project  risks  and  development  of  project  management  plans  (see  Table  6,  14  & 15)  which  influenced  their 
preliminary  design  (Table  6,  12). 

Table  6:  Judges  Ratings  for  Case  Response 

Ratings  for  individual  items  Averaged  across  Teams 


1 . The  team  identified  ail  of  the  important  issues  in  the  case. 

2.  The  team  effectively  addressed  ail  of  the  important  issues  in  the  case. 

3.  The  team  defined  the  perspectives  of  all  the  relevant  stakeholders  in  this  case,  (e  g.,  artists,  technicians, 

supervisors,  administration,  trainers,  organization.) 

4.  The  team  effectively  responded  to  all  of  the  relevant  perspectives  in  this  case. 

5.  The  team  effectively  analyzed  the  needs  identified. 

6.  The  team  identified  appropriate  alternative  solutions  for  each  need. 

7.  The  team  developed  an  instructional  goal  that  was  appropriate  for  the  case. 

8.  The  team  recommended  an  appropriate  overall  solution. 

9.  The  team's  specifications  for  personnel  to  be  involved  in  the  solution  were  appropriate. 

1 0.  The  team  effectively  integrated  relevant  professional  knowledge  (theories  and  practices)  into  their 
response. 

1 1 . Overall,  the  team’s  Needs  Assessment  and  Preliminary  Design  were  appropriate  for  the  case. 

12.  Overall,  the  team's  Needs  Assessment  and  Preliminary  Design  demonstrated  excellence. 

13.  The  team  was  proactive  in  making  recommendations  and/or  modifying  the  environment,  as  opposed  to 
being  only  reactive  and  giving  the  client  what  they  said  they  wanted. 

14.  The  team  identified  all  major  project  risks  and  developed  plans  to  manage  them. 

15.  The  team  presented  an  appropriate  quality  management  plan. 
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In  line  with  our  previous  study,  the  officials  we  interviewed  presented  a range  of  perceptions  about  the 
relationship  between  the  Teams’  performance  in  the  case  competition  and  their  design  expertise.  Two  officials  felt 
the  Teams’  ability  to  analyze  the  case  reflected  the  philosophy  of  the  University  IT  program  the  student  team 
represented: 


I think  some  might  be  in  a more  theoretical  program  so  that’s  why  in  my  eyes  they  may  have  missed  some  of  the 
practical,  real  world  things  that  were  going  to  happen.  Others  seemed  like  they  had  a feel  for  both,  the  theory  and 
practice  together. 
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I think  at  each  of  our  universities,  we  tend  to  have  a certain  style  that  we  impart  to  our  students  and  1 think  it  was 
very  evident  they  were  all  from  different  universities...  Some  took  a more  structured  approach,  and  others  took  a 
more  casual,  more  open-ended  approach  in  their  analysis. 

Five  of  the  officials  agreed  that,  although  the  Teams  reflected  a working  knowledge  of  the  ID  process  and 
systematically  conducted  the  needs  assessment,  they  demonstrated  a wide  range  of  real  world  ID  experience: 

Every  single  team  used  traditional  analytical  ID  approach  to  try  to  break  the  problem  down...  Yet,  1 personally  felt 
that  we  saw  along  a continuum  from  the  winning  response  to  the  weakest  response,  an  increasing  distance  from 
experience  in  the  real  corporate  world. 

The  students  felt  it  was  their  experience  applying  the  ID  process  that  affected  performance.  Six  of  the  seven 
Teams  felt  they  had  a strong  background  in  ID  theory  and  the  seventh  team  noted  additional  confidence  in  applying 
the  ID  process  through  their  prior  work  experience.  Several  respondents  felt  prior  experience  enhanced  their 
performance  and  continued  to  develop  during  the  case  analysis: 

I felt  our  team  achieved  a very  high  level  of  interpersonal  relations  and  the  use  of  instructional  design  theory.  And. 
the  use  of  relevant  experience...  All  those  were  well  applied  in  this  case  and  1 think  we  developed  it  even  further 
during  the  process  of  working  on  this  case. 

One  official  felt  that,  in  addition  to  their  understanding  of  the  theory  behind  instructional  design,  most  of  the 
participants  understood  what  an  EPSS  is.  However,  their  treatment  of  EPSS  was  limited  to  the  surface  features  and 
did  not  consider  the  potential  risks  commenting,  "which  is  not  surprising,  because  that’s  not  in  the  literature  yet.” 

Three  of  the  officials  felt  the  Teams  approached  the  needs  assessment  as  a novice  would,  accepting  the 
information  presented  at  "face  value."  They  commented  that  only  two  Teams  went  beyond  the  general  needs 
assessment  and  had  an  awareness  of  the  techniques  they  might  use  to  obtain  additional  information  so  their  needs 
assessment  would  be  more  thorough.  One  official  reflected  that  an  expert  would  not  automatically  pursue  the 
solution  requested  by  the  client  as  many  of  the  Teams  did: 

I think  they  did  a really  good  job  as  beginning  people.  The  solution  to  the  client  should  have  been  to  go  back  and 
do  more  of  a needs  assessment  and  not  go  into  training  at  this  point.  But.  I think  that  is  experience.  In  my  first  five 
years,  I would  have  pushed  for  training.  It  took  a lot  more  experience,  sophistication,  and  maturity  to  say  to  a 
client,  'Hey,  I know  what  you  want  to  have,  but  we  are  not  ready  to  do  this.' 

One  official  felt  the  students  exhibited  a degree  of  inexperience  in  their  proposed  solutions  as  well, 
suggesting  that  novices  tend  to  rush  and  throw  many  solutions  at  the  problem  simultaneously: 

They  were  throwing  so  many  solutions  at  the  clients  at  one  time  that  the  client  would  have  said  this  is  just  too 
much.  This  is  not  us..  Whereas  going  in  and  fixing  a couple  of  things  and  explaining  to  the  client  that  this  is  a life- 
long change  here  and  will  take  time  to  implement.  A sign  of  youth  is  wanting  to  go  in  and  fix  the  world  . . . It  is  just  a 
matter  of  learning  and  patience  that  comes  with  more  experience. 

Four  of  the  officials  thought  the  Teams  exhibited  inexperience  with  the  treatment  of  performance  support. 
One  official  felt  most  of  the  Teams  neglected  consideration  of  quality  management  issues,  an  important  factor  when 
designing  an  EPSS: 

Not  surprising  for  novice  performers  versus  expert  performers.  They  weren’t  taking  into  account  the  risk 
management  issues  involved  in  the  case...  They  kind-of  assumed  they  would  wave  a magic  wand  and  the  EPSS 
would  happen  and  that  it  would  happen  well.  They  tended  to  underrate  the  large  degree  of  re-do  found  in  the 
organization  as  a quality  management  issue. .. 

Respondents  from  most  of  the  Teams  felt  they  were  challenged,  as  well,  addressing  the  EPSS  issues. 

I was  definitely  challenged  in  the  EPSS  area  never  having  ever  worked  on  an  EPSS  and  only  doing  the  research  to 
prepare  for  the  case.  So,  I wasn’t  sure  how  those  really  worked.  I’ve  never  seen  one. 

I was  least  prepared  definitely  for  the  whole  kind  of  idea  of  EPSS.  You  know,  we  had  gone  over  general  kind  of 
performance  support  systems  but  not  specifically  electronic.  I wasn't  experienced  with  that  at  all,  so  I actually 
learned  a lot  about  that. 


Quality  of  the  Case 

When  we  spoke  with  the  students  about  the  degree  of  realism  in  the  case,  we  asked  if  they  thought  it  was 
sufficiently  compelling  so  they  could  enter  the  case  and  explore  the  ID  issues.  Most  of  them  commented  that  it  gave 
them  the  impression  that  it  was  real  enough  for  them  to  take  on  the  roles  of  the  characters: 
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It  was  realistic.  In  fact,  I must  say  that  while  I was  doing  it,  I was  convinced  that  there  had  actually  been  someone 
who  had  been  in  a similar  experience  and  type  of  industry.  That  is  actually  the  way  I thought  about  the  case. 

The  way  you  put  it  in  a real  world  setting  with  somebody  walking  into  a brand  new  company,  and  some  of  the 
other  little  things  that  you  talked  about  to  set  the  case  up,  were  kind  of  enjoyable  to  read  rather  than  just  reading 
some  dry  facts.  And,  again,  it  was  very  clear  and  very  straightforward  to  follow  all  the  way  through.  The 
interviews  were  well  written  and  really  helped  us  focus  on  what  the  issues  were  and  some  of  the  solutions  we 
decided  on. 

You  really  took  the  time  to  develop  personalities.  We  understood  where  people  were  coming  from...  I found  myself 
actually  sort  of  getting  into  each  character  and  sometimes  even  thinking  about  what  else  they  might  say... 

One  student  commented  that  the  case  was  not  realistic  because  the  animation  industry  is  foreign  to  her.  She 
did  find,  however,  the  personalities  of  the  characters  to  be  believable: 

There  aren’t  very  many  people  who  create  animation  and  certainly  none  in  my  world,  so,  only  like  reading  science 
fiction.  That  can  be  compelling,  but,  in  terms  of  real,  no.  What  I thought  was  good  was  there  were  a lot  of  different 
human  elements  in  there;  the  different  motivations  and  perspectives.  I thought  that  was  realistic. 

Most  of  the  officials  found  the  work  environment  in  the  case  to  be  extremely  realistic: 

The  organizational  context  issues  I thought  were  very  reflective  of  the  kinds  of  things  you  would  see  in  a real 
organization.  The  details  concerning  the  description  of  the  work  flow  and  the  description  of  the  tasks  as  far  as  they 
went  were  very  plausible... 

The  way  the  interviews  were  written,  the  collection  of  interviews,  they  were  presented  in  dialogue  form  that  made 
them  both  seem  very  realistic  and  sort  of  present  tense  if  that  makes  any  sense.  It  clearly  was  not  an  abstraction  of 
interviews,  but  it  was  presenting  the  actual  interview  and  I thought  that  aided  the  realism. 

When  you  have  the  two  groups  (characters  in  the  case),  particularly  the  older  experienced  people  with  the  less 
technological  background  and  the  newer  techie  group;  that  certainly  is  very  real  in  any  kind  of  setting.  I also  saw 
the  kinds  of  problems  that  really  didn’t  matter  what  your  industry  was  or  whether  it  was  industry  or  education.  I 
mean  this  is  a very  realistic  situation  that  everybody  faces. 

One  official  felt  it  helped  fill  a gap  in  instructional  design  training: 

There  was  not  just  an  obvious  find,  one  or  two  answers.  There  was  a provision  for  alternate  agendas  and  politics 
and  this  kind  of  stuff  is  exactly  what  I have  felt  is  lacking  in  an  awful  lot  of  instructional  design  training... 

This  official  also  commented  that  he  found  one  aspect  of  the  case  only  mildly  realistic,  and  that  was  the 
amount  of  responsibility  given  to  a junior  hire.  A second  official  began  to  feel  the  case  had  been  contrived  as  he 
observed  the  team  moving  further  into  it,  but  this  did  not  affect  the  students  engagement  in  the  process  and  they 
became  a part  of  the  story: 

There  were  times  when  all  of  a sudden  the  thing  seemed  like  a real-life  project  and  the  team  was  so  involved  in  it 
that  you  wouldn't  know  they  weren’t  working  for  the  company... 

Materials  and  Management  of  the  Case 

When  we  spoke  to  the  students  about  the  interface  of  the  Web-based  case,  we  were  interested  in  identifying 
whether  the  navigational  design  was  efficient.  We  also  wanted  to  know  whether  our  increased  focus  on  textual 
character  development,  in  place  of  audio  and  video  clips,  was  effective  in  providing  an  environment  that  was  easy  to 
access,  yet  deep  enough  to  present  the  complexities  of  the  real  world.  Most  of  the  students  interviewed  felt  the 
materials  at  the  site  were  easy  to  access  and  reflected  that  the  writing  style  helped  them  to  analyze  the  case: 

1 found  that  it  was  easier  to  process  the  case  because  of  the  way  it  was  not  only  written,  but  the  way  you  could 
access  things.  For  example,  the  fact  that  each  interview  was  a separate  component  on  the  web...  you  could  read 
what  Carl  Durersaid.  You  could  chronologically  look  at  what  happened  to  Jason  on  the  day  he  was  hired.  So,  for 
content  purposes... and  for  chronological  purposes,  it  was  easy  to  process  and  you  could  do  things  simultaneously 
by  clicking  back  and  forth. 

I think  the  way  that  the  interviews  in  the  case  were  organized  with  the  brief  intro... made  it  more  comprehensible. 

You  got  the  whole  picture  in  a very,  I wouldn’t  necessarily  say  concise,  but  in  such  an  organized  manner,  that  we 
were  able  to  understand  it  properly  and  get  a picture  in  our  heads.  I think,  if  you  can  visualize  things,  it  makes  it  a 
lot  easier. 

Respondents  who  had  read  previous  Case  Event  cases  thought  the  shift  in  focus  from  multimedia  to 
character  development  improved  the  quality  of  the  case: 
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I think  it  was  a very  professionally  done  job.  Your  web  site  looks  better  and  better  every  year.  The  case  responses 
are  getting  easier  to  read  and  more  understandable  every  year.  I think  your  learning  curve  has  really  gone  up  and 
you  are  putting  together  a classy  act  here. 

Well,  every  year  that  these  case  studies  have  come  out,  and  I have  read  all  of  them  since  you  all  started  this,  they 
are  getting  easier  to  read.  Really,  the  writing  that  you  guys  have,  the  style  that  you  use  to  build  these  cases,  really 
lends  itself  to  picking  up  the  ideas  that  you  want  out  of  the  case  very  easily. 

A student,  who  was  new  to  instructional  design  theory,  found  it  difficult  to  identify  the  problem  within  the 
complex  layers  of  the  story:  "It  was  a huge  packet  for  us  to  digest  in  the  short  period  of  time.  I would  highly 
recommend  that  it  be  more  compact  and  not  as  technical,  or  else  make  sure  that  first- year  students  are  not  allowed  in. 
It  was  a real  mismatch  to  try  to  put  it  all  together  and  learn  everything  as  we  go."  Another  student  from  this  team 
commented:  "While  this  was  frustrating,  I think  it  mirrors  the  actual  process  in  a real-life  situation." 

The  graphic  interface  for  the  Web  pages  was  designed  so  the  case  could  be  effectively  viewed  in  versions  of 
browsers  that  had  been  released  within  the  past  year.  We  found,  however,  that  a few  of  the  respondents  were  using 
older  browsers  that  did  not  recognize  colored  table  cells.  Consequently,  these  students  had  trouble  reading  the  case 
on-line  because  the  light  colored  background  of  the  table  cells  did  not  display.  And  they  were  viewing  dark  text  on  a 
dark  background. 

A few  issues  emerged  concerning  the  coordination  of  the  case  event  involving  the  limitations  placed  on  the 
students  and  the  format  of  the  response.  The  time  limit  for  group  meetings  was  seen  as  difficult  by  some  of  the 
respondents.  One  student  felt  the  word  limit  was  enough  of  a time  constraint:  "Why  the  need  for  the  time  constraints? 
Isn't  the  2000  word  limit  for  the  initial  response  enough  of  a constraint?"  Another  respondent  suggested  increasing 
the  time  limit  to  match  a typical  workday:  "I  understand  the  need  for  time  constraints,  it  might  be  helpful  if  the  six 
hour  limit  could  be  expanded  upwards  by  a small  amount  such  as  8 hours,  a usual  work  day."  A third  respondent  felt 
that,  although  a challenge,  this  is  reflective  of  real  life:  "You  could  always  use  more  time,  but,  I think  given  the  case 
and  the  situation,  ...it  was  probably  proportionally  appropriate  to  real  life." 

Two  of  the  Teams  had  contrasting  views  concerning  the  format  of  the  case  response.  A member  from  one 
team  felt  that  the  restriction  to  develop  a text-only  response  enabled  them  to  focus  on  the  case  analysis  and  design. 
Another  team  felt  that  the  text-only  restriction  diminished  the  quality  of  the  response  they  could  produce.  They 
suggested  that  it  would  be  more  reflective  of  actual  practice  and  allow  for  greater  creativity  if  Teams  could  place 
their  responses  in  the  form  of  a report  on  the  Web,  including  tables,  graphics,  and  links  to  relevant  Web  resources. 

Relevance  to  Actual  Practice 

Eighty-nine  percent  (25)  of  the  student  survey  respondents  and  95%  (18)  of  those  interviewed  commented 
on  issues  presented  in  the  case  they  found  to  be  relevant  to  actual  practice  and  several  issues  immerged.  The  limited 
time  frame  for  producing  results  was  seen  as  a challenge  for  "managing  multiple  priorities  simultaneously"  and  this, 
in  turn,  fed  into  the  organizational  dynamics  an  instructional  designer  must  contend  with  in  a corporation: 

The  issues  surrounding  the  great  gaps  that  exist  between  the  younger,  "techie”  types  and  the  older,  more 
experienced  employees  were  particularly  important.  Being  able  to  provide  solutions  to  bridge  this  gap  was 
especially  relevant.  Also,  the  fast-paced  working  environment  in  this  particular  case  is  probably  very  reflective 
generally  of  work  environments  today  where  deadlines  are  too  short  for  the  amount  of  work  required. 

Perspectives  on  needs  assessment  ranged  from  those  who  had  limited  prior  experience  applying  the  ID 
process,  and  learned  "the  importance  of  the  environmental  assessment  and  how  very  complicated  that  can  be  as  part 
of  the  overall  picture..." 

It  gives  you  a whole  different  perspective  on  front-end  analysis  in  a real-world  type  setting  as  opposed  to  a 
classroom  situational-type  project  where  you’re  just  going  through  the  motions. 

...The  degree  to  which  you  can’t  rely  on  individual  interviews  and  the  need  for  a composite  whole  of  many 
different  interviews  became  clear  to  me.  It  helped  me  understand  that  you  can’t  just  jump  in  and  do  something. 

You  need  to  do  thorough,  thorough  needs  analysis.  You  need  to  analyze  everything,  consider  everything. 

We  don’t  really  get  a chance  to  do  as  many  needs  assessments,  I think,  as  we'd  like  to  do  before  we  leave  the 
program.  So,  I think  every  one  of  them  has  got  some  major  learning  issues  in  it.  The  more  we  do,  the  better  off  we 
are. 


Two  students  who  felt  they  were  well  grounded  in  design  theory  and  application,  yet  saw  new  challenges 
they  would  face  when  conducting  a needs  assessment  in  environments  similar  to  the  case: 

I think  the  other  part  that’s  relevant  to  the  practice  of  ID  is  the  Jason  Tilmans’,  which  we  all  may  be  in  the  future. . . 
Where,  you’re  told,  This  is  what  the  need  is.’  You  do  the  assessment  and,  albeit,  it’s  a need,  but  combined  with  a 
whole  bunch  of  other  stuff...  Then,  how  do  you  choose  to  link  the  need  that  you’ve  been  hired  to  address,  but  still 
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be  realistic?  I felt  this  was  very  germane  to  ID  practice  because  it's  rare  that  you  get,  ’Here's  the  need,  meet  it,'  and 
everything’s  hunky  dory. 

I blush  when  I think  about  how  I probably  analyzed  situations  at  work  before  doing  the  case.  Whether  they  were  ID 
or  just  performance  technology  issues,  I feel  like  they  were  limited  in  terms  of  how  I would  analyze  needs  or  how  I 
would  not  be  so  multi-perspective  as  I am  now.  I feel  that  critical  thinking  skills  and  my  writing  has  had  to  change 
to  really  document,  if  you  will,  with  evidence,  why  I believe  an  assumption  or  why  Tm  making  a statement.  I 
believe  this  has  grown  and,  for  that,  I'm  very  grateful,  and  I think  that’ll  serve  me  well  no  matter  what  I do. 

Most  of  the  students  interviewed  commented  on  emergent  issues  such  as  EPSS  and  how  they  must  be 
integrated  with  the  ID  process.  Some  of  the  students  went  on  to  express  how  their  experience  addressing  these  issues 
during  the  case  has  motivated  them  to  continue  learning  about  the  topic: 

We  thought  it  was  very  interesting.  In  fact,  the  whole  team  was  really  excited  about  it.  We  thought  that  the 
challenge  of  EPSS  was  relevant  to  things  that  are  going  on  right  now  in  a lot  of  industries... 

What’s  relevant,  is  that  EPSS  seems  to  be  an  emerging  trend  all  of  us  need  to  understand.  I spoke  to  some  of  my 
colleagues  in  California,  Silicon  Valley,  and  it’s  a hot  issue  out  there.  They’re  all  getting  involved  in  it.  This  is  very 
relevant.  I have  to  tell  you,  I was  very  grateful  that  the  case  brought  this  to  the  forefront  of  my  mind. 

Performance  support  is  absolutely  crucial  and  something  that  I am  going  to  explore  further  because  of  this  case 
study.  I think  these  skills  are  necessary  for  an  instructional  designer  to  work  with  a company  where  you  are  going 
to  have  to  increase  productivity,  justify  training,  and  work  with  EPSS  systems.  So,  my  reflection  is  that  this  really 
opened  my  eyes  to  a whole  different  world.  I think  I am  going  to  have  to  pursue  it  and  get  better  at  it  if  I am  going 
to  do  instructional  design... 

Although  most  of  the  students  commented  on  team  dynamics  when  reflecting  on  the  case  analysis  process, 
seven  of  them  remarked  upon  their  heightened  awareness  of  the  need  for  these  skills  in  actual  practice.  They 
compared  this  need  to  the  challenges  of  team  dynamics  presented  in  the  case: 

I mean,  just  taking  it  from  Jason’s  perspective,  I think  it  is  really  very  important  to  have  a good  understanding  of 
what  will  gain  support  and  develop  a team  that  you  can  work  with.  Not  that  he  necessarily  did  that,  or  we 
necessarily  suggested  it,  but  this  is  what  I learned  from  the  experience. 

I think  we  came  out  of  it  with  a really  increased  awareness  of  what  can  go  on:  The  experience  of  hurrying  through 
the  needs  analysis  and  linking  the  need  to  some  practical  issues  in  your  solution.  Then  writing  up  a formal  proposal 
and  possibly  presenting  it  to  a group  of  people  who  you  don’t  really  know... the  emotional  response  might  come 
back  with  things  that  you  hadn't  anticipated.  Also,  the  dynamics  of  working  in  a big  team  like  that  using  electronic 
collaboration. 

The  issues  were  secondary  to  the  personalities  involved.  Often  in  my  studies,  I forget  about  the  people  and  focus 
on  strategy  and  theory. 

Six  of  the  students  commented  on  the  need  to  consider  multiple  perspectives  and  solutions  as  well: 

In  the  process  of  identifying  key  issues  from  a complex  array  of  concerns  and  issues,  it  is  important  to  quickly 
identify  what  are  the  biggest  causes  or  factors.  Also,  being  able  to  identify  training  versus  non-training  issues  and, 
of  course,  working  with  people  is  always  a useful  skill  in  the  real  world. 

The  most  important  thing  about  this  case  was  when  we  first  jumped  into  it,  we  thought.  ’OK,  they  want  EPSS  as  a 
solution.’  And  it  helps  to  remember  that,  even  though  this  may  be  Jason’s  talent,  it  is  not  the  only  solution...  The 
good  thing  is  to  look  at  all  the  possible  solutions. 

The  officials  were  asked  to  reflect  on  how  this  experience  contributes  to  the  development  of  ID  students 
professional  skills.  Four  officials  commented  on  the  benefit  of  reading  all  of  the  case  responses  and  officials' 
comments  at  the  conclusion  of  the  event: 

If  you  read  someone  else's  perspective  and  response,  it  remains  in  the  back  of  your  mind  where  you  can  call  on  it 
when  you  encounter  that  type  of  situation  or  challenge  in  your  career. 

One  official  agreed  that  the  case  experience  provides  an  opportunity  to  practice  knowledge  and  skills  and 
get  feedback  on  the  results  from  several  resources.  However,  he  reflected  on  the  difficulty  in  determining  how  much 
the  case  experience  contributed  to  the  students'  development  of  their  professional  skills: 

I am  hesitating  only  because  1 think  it  is  going  to  be  difficult  to  tell  exactly  how  much  or  in  what  way.  They  got  to 
compare  what  they  came  up  with  against  other  students,  against  Judges,  against  expert  responses,  so  they  got  a lot  of 
feedback  from  several  different  sources  that  they  could  use  to  do  a self-assessment...  I've  just  got  to  believe  that  is  a 
growth  experience,  although  we  don't  know  anything  about  how  they  use  that  feedback. 


Another  official  noted  that  his  students  were  interested  in  learning  more  about  the  issues  that  were  identified 
in  the  case: 


They  wished  they  knew  more  about  some  of  the  issues  that  came  up  in  the  case  study,  for  example,  different  ways  to 
conduct  task  analyses.  Now  they  want  to  leam  that  in  the  class.  The  students  involved  in  this  project  would  jump  at  the 
chance  to  get  some  more  instructional  experiences  over  the  things  they  realized  they  didn’t  have  much  experience  in;  task 
analysis.  EPSS,  and  project  management. 


Value  of  Participation 

The  students  and  officials  were  asked  to  share  their  opinion  on  the  value  of  the  case  study  method  and  the 
case  competition  using  Likert  response  options  from  1 (strongly  disagree)  to  4 (strongly  agree).  The  students 
overwhelmingly  felt  that  the  use  of  the  case  study  method  is  valuable  for  developing  ID  expertise  {M  = 3.61,  SD  = 
0.50)  and  that  the  case  event  was  of  value  in  preparing  them  for  future  ID  projects  (M  = 3.57,  SD  = 0.50).  This 
perception  was  shared  by  the  officials  in  regard  to  the  value  of  the  case  study  method  (M  = 3.93,  SD  = 0.27)  and  the 
value  of  the  case  competition  in  preparing  ID  students  for  future  projects  {M  = 3.64,  SD  = 0.50). 

Several  of  the  students  and  officials  share  their  perspectives  on  the  value  of  the  case  experience.  Most  of  the 
students  were  positive,  describing  how  the  experience  enhanced  their  preparation  for  professional  practice: 

The  opportunity  to  solve  a problem  in  an  area  that  I knew  so  little  about  was  very  valuable.  I was  able  to  make  the 
connection  that  instructional  design  isn’t  always  for  stand-up  training  or  CBT...  This  is  by  far  the  best  learning 
experience  I have  had  in  my  entire  educational  career. 


and 

I just  read  in  the  newspaper  ads  last  Sunday  about  trainers  needed  with  skills  in  knowledge  management.  I thought, 
this  is  kind-of  neat  that  this  case  study  addressed  all  these  things  that  we  don’t  really  address  in  a classroom  setting 
or  in  some  work  settings,  like  the  one  I’m  in.  Normally,  or  before  this  case  study,  when  I thought  of  instructional 
design,  I thought  of  designing  instruction  for  stand-up  training  or  computer-based  training  or  self-based  training.  I 
never  really  had  the  context  to  put  it  in  a real-life  production  environment  and  it  just  kind-of  opened  my  eyes  to  a 
whole  new  use  for  this  skill  set. 

All  of  the  officials  who  responded  were  positive  and  shared  their  perspectives  on  what  was  most  valuable. 
There  were  several  enthusiastic  comments  about  the  value  of  the  case  competition  experience: 

I think  this  is  one  of  the  most  significant  events  that  has  occurred  in  their  ID  education... 

I thought  this  case  was  wonderful!  Having  spent  some  time  in  a Glen  Michener  type  of  role,  I fully  understood  the 
different  perspectives  and  felt  that  the  designers  of  this  case  were  thoughtful,  accurate  and  insightful! 

As  my  team  captain  said,  this  was  the  most  valuable  learning  experience  of  her  career.  What  made  it  so?  In  my 
opinion  the  following  elements  were  key:  Competition,  particularly  with  some  big-name  schools  and  people,  time 
limits,  word  limits,  and  the  high  quality  of  the  case  itself:  Well  written,  creative,  engaging,  relevant. 


and. .. 

All  the  theory  in  the  world  is  useless  unless  practitioners  know  their  roles  as  change  agents.  This  case  placed  the 
students  in  a situation  where  understanding  varying  perspectives  and  agendas  was  imperative.  They  also  had  to  be 
sensitive  to  the  repercussions  of  their  recommendations.  The  case  pivoted  on  this,  and,  as  such,  was  a great 
experience  for  the  students.  The  mere  fact  that  this  was  event  competitive  was  powerful  as  well. 

For  me,  the  most  valuable  part  was  seeing  perhaps  some  perspective  on  what  future  graduates  are  thinking  about 
bringing  their  design  skills  into  the  workplace.  I believe  that  there  are  some  extremely  clever  people  who  will  be 
professionals  in  our  field  soon,  but  I think  it’s  clear  that  we  have  some  ’’broadening”  to  do...  We  need  to  insure 
that  new  designers  and  future  decision-makers  in  our  field  are  equipped  with  both  the  design  skills  and 
interpersonal  skills  to  be  true  "change  agents”  in  their  careers. 

Of  the  six  officials  who  are  not  yet  using  ID  cases  in  their  classes,  three  commented  that  they  are 
considering  integrating  cases  into  their  courses  as  a result  of  this  experience: 

We  have  actually  talked  about  restructuring  the  two-semester  instructional  design  sequence  to  actually  take 
advantage  of  the  case  study...  because  it  contextualizes.  I mean  it  is  just  good  instructional  design...  It  comes  about 
as  close  to  situated  cognition  as  you  can  get  over  the  web... 


Discussion 

Sixty-seven  percent  of  the  participating  students  and  82%  of  the  officials  responded  to  the  follow-up  survey  and  43% 
of  the  participating  students  and  71%  of  the  officials  agreed  to  be  interviewed.  They  shared  their  perspectives  on  the 
value  of  the  Case  Event  as  a tool,  for  the  preparation  of  professional  practice  in  instructional  design.  The  students 
also  described  the  processes  in  which  they  engaged  to  analyze  the  case  and  create  a design  solution,  and  several  (8  of 
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the  14,  or  57%)  of  the  officials  commented  on  their  observations  of  this  process.  In  the  following  section,  we  discuss 
the  implications  of  our  findings  as  they  pertain  to  our  primary  question  guiding  this  study  and  follow  with  discussion 
of  our  secondary  questions. 

■ Is  a Web-based  case  competition  a worthwhile  medium  for  expanding  professional  knowledge  and  exploring  emerging 
issues  in  instructional  design? 

The  potential  to  expand  their  preparation  for  professional  practice  in  ED  was  noted  by  the  majority  of  the 
students  who  commented  (11  of  13,  or  85%)  on  their  reasons  for  participating  in  the  Case  Event.  Students  from  two 
of  the  Teams  that  were  constructed  as  a part  of  a course  observed  a lack  of  commitment  from  team  members  who 
remarked  they  had  been  required  to  participate.  At  the  conclusion  of  the  Case  Event,  however,  all  of  the  respondents 
noted  the  value  of  the  case  experience  as  a tool  for  developing  ID  expertise.  This  suggests  that,  although  most 
participants  value  the  ID  experience  gained  during  the  case  event,  those  who  participate  by  choice  are  more  likely  to 
engage  in  the  process  and  more  fully  benefit  from  the  experience. 

Consistent  with  our  previous  study,  the  participants  were  enthusiastic  about  the  competition  and  their 
engagement  in  the  process  began  prior  to  the  event.  All  but  one  student  read  at  least  one  of  the  previous  IT  cases  and 
the  majority  of  them  read  the  accompanying  case  responses  (85%)  and  expert  (71%)  critiques.  Prior  experience 
using  ED  cases  as  a learning  tool  had  increased  since  our  previous  study  from  only  one  student  to  almost  one  third  of 
the  students  (28%).  In  addition  to  reviewing  the  materials  available  on  the  Web  site,  other  preparation  included 
research  of  learning  theory,  review  of  instructional  design  literature,  and  procedures  for  collaboration.  Just  under  one 
third  of  the  students  we  interviewed  indicated  they  had  reviewed  the  recommended  articles  on  EPSS  as  well. 

The  case  analysis  and  response  generation  described  in  this  paper  reinforces  the  findings  from  our  previous 
study.  There  were  several  common  aspects  of  coordination  exhibited  by  most  of  the  Teams. 

This  included: 

• reading  the  case  prior  to  the  first  meeting; 

• meeting  two  or  three  times; 

• using  the  first  meeting  to  share  perspectives  and  develop  a plan  for  the  response  and  the  second  meeting  to  generate  a 
solution; 

• designation  of  one  or  two  individuals  to  compile  ideas  into  a single  document;  and  circulation  of  the  document  among 
team  members  for  review. 

Some  Teams  divided  the  case  response  among  team  members  according  to  the  competition  guidelines  while 
others  completed  all  of  the  sections  together  or  in  sub-groups.  One  Team  was  unique  in  its  approach,  completing 
individual  analyses  and  conducting  research  before  their  first  meeting. 

Some  significant  new  aspects  that  emerged  from  the  analysis  and  response  generation  pertained  to 
collaboration.  All  of  the  Teams  perceived  collaboration  as  important  to  productivity  and  there  were  various  factors 
influencing  team  dynamics.  Several  participants  felt  prior  experience  (or  lack  thereof)  working  together  contributed 
to  their  effectiveness  as  a team.  Those  with  prior  experience  had  few  disagreements,  while  others  felt  that  lack  of 
experience  working  in  large  groups  diminished  the  effectiveness  of  their  meetings,  in  particular  sharing  multiple 
perspectives  and  coming  to  a consensus  as  a whole  group.  Some  Teams  addressed  this  challenge  by  splitting  into 
sub-groups,  which  was  seen  by  some  participants  as  productive  and  others  as  a poor  use  of  time  due  to  the 
redundancy  in  the  work  produced  in  the  sub-groups.  Leadership  was  another  factor  that  emerged  as  an  important 
aspect  of  team  collaboration.  Some  participants  felt  that  shared  responsibility  was  effective,  while  others  felt  it 
resulted  in  varying  levels  of  commitment  to  the  project.  In  spite  of  these  challenges  in  collaboration,  the  majority  of 
the  respondents  (79%)  felt  that  the  team  process  was  one  of  the  most  valuable  learning  experiences  in  this  activity. 
Slavin’s  (1995a,  1995b)  research  on  group  goals  and  individual  accountability  is  reflected  in  the  team  that 
successfully  collaborated  due  to  a combination  of  factors.  These  included:  small  group  size,  a designated  leader  and 
action  plan,  individual  generation  of  responses,  and  meetings  reserved  for  the  integration  of  ideas  and  decision 
making. 

Students  who  volunteered  to  participate  in  the  case  event  felt  the  competition  was  an  important  component 
in  the  case  experience,  increasing  the  sense  of  challenge  and  motivating  them  to  participate.  Most  of  these  students 
also  viewed  the  prospect  of  being  judged  as  motivating.  They  were  compelled  to  research  beyond  the  given 
information  in  the  case.  One  team  who  volunteered,  however,  described  a negative  influence  on  their  performance. 
They  reflected  their  efforts  to  "second-guess"  the  Judges  throughout  the  generation  of  the  case  response  and 
commented  that  it  added  stress  to  their  learning  experience.  The  majority  of  the  officials  who  commented  on  the 
value  of  competition  agreed  that  it  enhances  the  learning  experience  by  engaging  students  cognitively  and 
emotionally.  There  were  concerns  however,  that  the  challenge  to  compete  with  doctoral  and  experienced  ID  students 
could  be  intimidating  for  beginning  master's  level  students.  This  sentiment  was  voiced  by  the  team  composed  of 
novice  ID  students  who  felt  they  were  not  prepared  to  compete  without  a foundation  in  the  ED  process. 
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As  in  our  previous  study,  the  Teams  performed  well  in  their  case  responses.  According  to  the  ratings  on  the 
overall  performance  item,  the  Judges  indicated  agreement  or  strong  agreement  that  team  performance  was  excellent 
with  the  exception  of  one  team  that  received  a lower  rating  from  two  Judges.  Ratings  on  the  specific  criteria  suggest 
that,  in  general,  the  Judges  felt  positive  about  the  Teams'  application  of  ID  theory  and  demonstration  of  insight  into 
ID  practice.  Their  weakest  performance  was  in  the  identification  of  project  risks  and  the  development  of  project 
management  plans. 

The  Judges  related  several  observations  on  team  performance  and  design  expertise  to  the  differences 
between  novice  and  expert  designers,  similar  to  those  identified  by  Rowland  (1992).  For  instance,  the  case  responses 
reflected  the  approach  taken  by  novice  designers  who  accept  information  at  face  value  and  move  immediately  to  the 
generation  of  solutions.  Most  of  the  Teams  pursued  the  solution  requested  by  the  client  in  the  case  and  were 
observed  to  throw  many  solutions  at  the  problem  simultaneously.  In  contrast,  experts  will  draw  from  previous 
experience  and  reflect  on  a wide  range  of  factors  as  they  interpret  the  design  problem  (Perez  & Emery,  1995; 
Rowland,  1992).  This  was  evident  in  two  Teams  who  went  beyond  the  case's  general  needs  assessment  and 
demonstrated  an  awareness  of  techniques  they  might  use  to  obtain  additional  information.  Students  recognized  the 
effects  of  prior  experience  on  their  performance  as  well,  noting  that  their  expertise  applying  the  ID  process  and 
interacting  with  team  members  was  evident  in  the  quality  of  their  case  response.  Some  students  commented  that  this 
expertise  continued  to  develop  during  the  case  experience. 

Embedding  an  emergent  issue  in  ID  practice,  EPSS,  into  the  case  study  was  seen  as  one  of  the  most 
challenging  and  valuable  aspects  of  the  case  experience.  Most  of  the  participants  remarked  that  the  case  study 
introduced  them  to  EPSS  and  to  the  necessity  of  learning  how  to  integrate  emergent  issues  into  the  ID  process.  They 
learned  that  instructional  design  goes  beyond  the  scope  of  training  and  CBT  (computer  based  design).  Some  of  the 
officials  felt  that,  all  of  the  Teams  performed  well  as  beginners;  they  were  able  to  address  the  surface  issues  in  their 
treatment  of  EPSS.  Most  of  them  neglected  consideration  of  important  components  of  EPSS  design  such  as  quality 
management  issues,  including  risk  management.  This  was  considered  to  be  another  example  of  novice  versus  expert 
performers.  Some  of  the  students  commented  that  they  were  encouraged  to  continue  learning  about  the  topic  while 
others  noted  they  had  become  aware  of  its  prominence  in  the  field  of  ID  and  were  pleased  they  had  been  introduced 
to  the  topic. 

All  of  the  responding  officials  and  students  agreed  or  strongly  agreed  that  the  Case  Event  experience  had 
been  valuable  for  developing  ID  expertise  and  enhanced  their  preparation  for  professional  practice.  One  official 
noted  that  the  Case  Event  helped  the  students  recognize  their  role  as  change  agents  by  placing  them  in  a situation 
where  they  were  forced  to  consider  multiple  perspectives  and  the  consequences  of  their  recommendations.  Officials 
and  students  alike  felt  that  the  case  competition  was  one  of  the  best  experiences  that  had  occurred  in  their  ID 
education.  Finally,  one  official  noted  the  value  of  the  case  competition  for  IT  programs,  providing  them  a means  to 
measure  what  design  skills  graduates  are  bringing  into  the  workplace. 

• Did  this  case  experience  alter  student  perceptions  of  skills  they  need  for  professional  practice  in  ID? 

When  asked  what  they  had  learned  relevant  to  actual  practice,  students  noted  that  their  awareness  of  what 
knowledge  and  skills  they  will  need  for  actual  practice  goes  beyond  the  theory  and  models  of  ID  they  practice  in  the 
classroom.  The  degree  of  knowledge  and  skills  recognized  correlated  to  the  students’  prior  experience  in  the  field  of 
ID.  Those  with  limited  experience  applying  the  ID  process  gained  a deeper  understanding  of  components  within  the 
needs  such  as  the  value  of  the  environmental  assessment  and  the  necessity  for  obtaining  multiple  perspectives 
through  multiple  interviews.  One  of  the  Teams  inquired  into  studying  the  instructional  design  issues  they  had 
identified  in  the  case  with  their  team  sponsor.  Students  who  entered  the  competition  with  some  prior  experience 
practicing  ID  drew  a correlation  between  the  case  experience  and  the  new  challenges  they  would  face  in  actual 
projects  that  might  occur  in  environments  similar  to  the  case.  Most  notable  was  the  necessity  to  support  all 
assumptions  and  design  solutions  with  evidence.  Both  beginning  and  advanced  students  recognized  the  need  to 
continue  developing  interpersonal  skills  which  are  essential  to  successful  practice  in  ID.  Officials  saw  a valuable 
opportunity  for  the  students  to  continue  to  expanding  their  professional  knowledge,  as  well,  by  exploring  the 
collection  of  IT  cases,  expert  responses,  and  feedback  from  the  Judges. 

• Did  the  design  of  our  Web-based  case  enhance  the  sense  of  realism  and  complexity  necessary  to  compel  exploration  of  ID 
issues  and  practice? 

We  focused  the  design  of  our  case  on  character  development  and  designed  the  interface  to  match  its  theme. 
Our  results  suggest  our  emphasis  on  character  development  and  the  supporting  interface  design  helped  enhance  the 
sense  of  realism:  All  but  one  respondent  felt  the  case  was  realistic  and  sufficiently  compelling  for  exploration  into 
the  ID  issues.  The  disagreeing  student  (who  was  from  the  beginning  ID  class)  felt  limited  by  her  lack  of  knowledge 
about  the  animation  industry  and  found  it  too  complex  for  entry  level  students  in  ID.  She  did,  however,  find  the 
characters  to  be  believable  and  representative  of  personalities  she  has  interacted  with  in  other  work  environments.  A 


facilitator-guided  analysis  of  the  cases  may  be  a better  suited  learning  experience  for  beginning  ID  students  who  do 
not  yet  have  the  background  knowledge  of  ID  theory  necessary  to  solve  the  case  problems  on  their  own.  According 
to  Vygotsky  (1978),  expert  facilitators  can  build  on  the  learners'  existing  knowledge  by  providing  guidance  and 
feedback,  which  will  help  them  move  beyond  their  current  level  of  competence.  As  a result,  they  can  begin  to  solve 
problems  as  does  someone  with  expert  experience. 

Those  who  commented  on  the  sense  of  realism  reflected  in  the  case  study  felt  that  character  development 
and  case  format  helped  them  to  focus  on  the  issues  in  the  case.  Some  noted  they  had  become  immersed  in  the  story, 
taking  on  the  role  of  each  character,  and  were  able  to  imagine  how  the  personalities  might  answer  their  questions. 
Most  of  the  officials  found  the  case  extremely  realistic  as  well,  commenting  that  the  key  issues  represented  the  types 
of  problems  one  would  find  in  any  setting.  Other  aspects  of  the  case  contributing  to  the  sense  of  realism  included  the 
context  of  the  company,  the  design  of  the  organization  charts,  and  the  presentation  of  the  interviews  in  dialogue 
form.  Several  officials  and  students  commented  that  these  elements  helped  the  reader  experience  the  case  through  the 
eyes  of  the  characters. 

Limitations  of  these  findings 

It  is  possible  that  students  who  did  not  respond  may  have  had  different  perceptions  of  the  value  of  the  case 
experience  as  a learning  tool  in  instructional  design.  Participants  received  three  email  notices  about  the  survey  and 
their  Team  Sponsors  were  asked  to  announce  it  to  the  team  as  well.  Survey  respondents  were  sent  individual  e-mail 
requests  for  interviews.  We  feel  that  all  who  wanted  to  respond  had  the  opportunity  to  do  so. 

This  paper  presents  the  experience  of  participants  in  a case  competition.  Therefore,  we  cannot  assume  the 
level  of  enthusiasm  and  motivation  exhibited  during  this  activity  are  representative  of  case  methods  employed  during 
a regular  ID  class. 

We  know  that  the  participants  feel  they  gained  skills  for  actual  practice  from  this  experience.  We  do  not 
know  however,  whether  the  students  will  retain  their  enthusiasm  and  apply  what  they  learned  in  other  settings.  As 
one  official  noted,  with  the  wealth  of  feedback  the  participants  get  through  the  multiple  responses  from  the 
participants,  experts,  and  judges,  we  can  assume  this  was  a growth  experience.  We  do  not  know,  however,  hpw  the 
students  will  use  this  feedback.  Research  of  long-term  outcomes  can  help  us  identify  whether  students  successfully 
integrate  what  they  learned  during  this  experience  into  instructional  design  projects,  teamwork  or  project 
management  design. 

Future  research  and  development  directions 

As  the  use  of  cases  in  instructional  design  classrooms  increases,  so  will  the  number  of  students  participating 
in  the  case  competition  who  have  prior  experience  analyzing  ID  cases.  Future  research  of  Case  Event  outcomes  can 
help  us  identify  whether  these  students  are  transferring  what  they  learned  in  prior  cases  to  the  new  environments 
represented  in  the  competition  cases.  This  may  provide  further  insight  into  the  effectiveness  of  case  studies  in 
narrowing  the  gap  between  novice  and  expert  practice  of  instructional  design. 

While  most  of  the  respondents  considered  the  competition  and  judging  valuable  aspects  of  the  case  event, 
some  of  the  students  were  intimidated  by  the  prospect  of  being  judged.  Some  students  tried  to  emulate  what  they 
thought  the  Judges  would  consider  the  "correct"  response  rather  than  using  their  own  perceptions  to  design  what  they 
believed  to  be  the  best  possible  solution.  This  may  have  diminished  the  effectiveness  of  the  learning  experience  and 
we  would  like  to  know  what  factors  contribute  to  this  reaction  to  the  competition  environment.  Future  research  can 
also  tell  us  whether  the  judging  process  diminishes  the  real  world  experience  for  these  students. 

Volunteers? 

For  the  past  three  years,  we  have  facilitated  the  IT  Case  Event,  writing  and  producing  Web-based  case 
studies  and  coordinating  the  competition.  To  encourage  collaboration  between  IT  programs  across  the  country,  we 
would  like  to  extend  an  invitation  to  other  universities  to  take  on  this  role  or  coordinate  with  some  other  institutions 
in  facilitating  the  competition.  Each  university  has  unique  perspectives  on  application  and  theory  in  ID  and  we  feel 
facilitation  of  the  event  from  multiple  universities  can  enhance  the  case  experience  for  all  of  the  participants. 
Implications  for  the  training  of  instructional  designers 

We  continue  to  find  that  the  cases  are  a useful  tool  for  encouraging  students  to  explore  ID  issues.  Students 
who  participate  out  of  choice  will  gain  more  from  the  experience  than  those  who  are  required  to  participate.  Limiting 
group  size  to  a maximum  of  4 or  5 persons  can  provide  the  best  dynamics  for  discussion  and  consensus.  Web-based 
case  methods;  students  enthusiastically  engage  in  the  case  analysis  and  feel  that  it  helps  prepare  students  for 
professional  practice  in  instructional  design.  Cases  cannot  replace  actual  experience,  yet  they  can  expand  the 
learners'  depth  of  understanding  applying  the  design  process.  The  experience  in  solving  problems  in  the  multiple 
environments  presented  in  the  case  studies  can  increase  the  novice  designer's  arsenal  of  design  tools.  The 
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competition  is  motivating  and,  when  implemented  in  a collaboration  environment,  the  case  experience  provides  a 
valuable  venue  for  expanding  professional  knowledge. 
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The  Instructional  Technology  Clinical  Experience:  Expectations  and  Realities 


Katherine  S.  Cennamo 
Glen  Holmes 
Virginia  Tech 


"To  learn  to  use  tools  as  practitioners  use  them,  a student,  like  an  apprentice  must  enter  that  community  and  its 
culture . Thus  in  a significant  way  learning  is,  we  believe,  a process  of  enculturation. " {Brown,  Collins,  & 

Duguid,  1989,  p.  33) 

Graduate  schools  in  Instructional  Technology  (IT)  typically  prepare  students  with  the  tools  of  instructional 
design  and  development.  IT  programs  usually  require  classes  in  instructional  design,  learning  and  instructional 
theories,  and  technical  production  skills.  However,  studies  of  expert  performance  indicate  that  experts  in  a domain 
possess  more  than  knowledge  of  subject  matter  specific  facts,  concepts  and  procedures.  Experts  also  posses  heuristic 
strategies  which  include  general  techniques  and  "tricks  of  the  trade”,  control  strategies  which  direct  the  process  of 
carrying  out  a task,  and  learning  strategies  that  guide  the  expert  in  learning  new  concepts,  facts,  and  procedures. 
Research  in  the  development  of  expertise  suggest  that  these  additional  skills  may  develop  in  situations  that  involve 
solving  complex  ill-defined  problems  under  the  guidance  of  a more  capable  other  (Brown,  et.  al.,  1989).  Thus,  the 
skills  of  an  expert  may  be  developed  through  immersion  in  the  culture  of  practice,  unfortunately,  the  only  experience 
many  graduate  students  have  in  the  design  and  development  of  instructional  materials  is  through  small  projects  in 
courses;  they  may  leave  school  with  little  experience  in  the  design  and  development  of  complex  projects  for  real 
clients,  and  with  few  opportunities  for  developing  complex  skills  through  enculturation. 

Several  methods  have  been  suggested  to  involve  graduate  students  more  extensively  in  the  entire  design  and 
development  enterprise.  Specific  methods  include;  the  preparation  and  examination  of  case  studies;  role  playing  and 
simulations;  instructional  design  studios;  class  work  on  real  projects;  internships,  apprenticeships,  public 
presentations,  competitions  among  graduate  students;  examination  of  design  artifacts,  and  analysis  of  examples  of 
expert  work  (Rowland,  1993;  Rowland,  Fixl,  & Yung,  1992;  Quinn,  1994;  Tripp,  1994). 

We  developed  the  idea  of  an  Instructional  Technology  “Clinical”  to  address  the  needs  of  our  students  for 
immersion  in  the  practice  of  instructional  design  and  development.  We  envisioned  an  experience  based  on  an 
apprenticeship  model  of  teaching  and  learning  that  included  an  instructional  design  studio  experience  (the  “Clinic 
course”),  public  presentation  of  student  work  (as  part  of  student  evaluation),  and  the  examination  and  development 
of  case  studies  and  design  artifacts  (as  an  orientation  to  projects  and  as  part  of  student  evaluation). 

We  explored  the  perceived  need  for  a “Clinical  experience”  to  facilitate  learning  through  immersion  in  “real 
problems”  that  provide  opportunities  for  participants  to  apply  program  ideas  and  techniques.  During  the  fall  1997 
semester,  27  students  enrolled  in  our  Instructional  Technology  Professional  Seminar  participated  in  a discussion  of 
the  Clinical  concept.  We  followed  the  period  of  open  discussion  with  a questionnaire  asking  students  to  identify  the 
benefits  of  such  an  experience,  their  related  concerns,  and  ideas  for  implementation.  We  developed  a specific  plan 
for  an  “IT  Clinical”  based  on  our  examination  of  student’s  perceptions  and  review  of  literature.  In  this  paper,  we 
will  provide  an  overview  of  the  Clinical  course  and  compare  the  perceived  “reality”  of  the  Clinical  experience,  in 
terms  of  benefits  and  concerns,  with  students’  initial  expectations  of  the  Clinical  experience. 

The  IT  Clinical  Plan 

" Cognitive  apprenticeship  methods  try  to  enculturate  students  into  authentic  practices  through 
activity  and  social  interaction  in  a way  similar  to  that  evident-  and  evidently  successful-  in  craft 
apprenticeship . " (Brown,  et.  al .,  1989,  p.  37) 

Collins,  Brown  and  Holum  (1991)  outline  a framework  for  designing  cognitive  apprenticeship  learning 
environments.  Their  framework  suggests  an  appropriate  sociology  for  effective  cognitive  apprenticeship  learning 
environments,  methods  for  developing  expertise,  types  of  content  knowledge  required  for  expertise,  and  how  to 
sequence  learning  activities.  Based  on  this  model  of  cognitive  apprenticeship,  we  have  incorporated  each  of  the 
proposed  pedagogical  features  into  the  Clinical  experience. 

Sociology 

The  sociology  of  the  learning  environment  includes  the  context  and  social  environment  in  which  the 
learning  occurs.  Collins,  et.  al.  (1991)  suggests  that  the  learning  environments  should  establish  a social  context 
where  learning  is  situated,  students  participate  in  a community  of  practice,  students  are  intrinsically  motivated,  and 
cooperation  is  encouraged. 
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Situated  learning 

Situated  learning  is  defined  as  "learning  knowledge  and  skills  in  contexts  that  reflect  the  way  knowledge 
will  be  useful  in  real  life"  (Collins,  1991,  p.  122).  The  purpose  of  the  Clinical  experience  is  to  provide  an 
experiential  learning  environment  where  students  apply  program  ideas  and  techniques  to  develop  instructional 
materials  for  the  college,  university  community,  and  external  organizations.  Projects  with  actual  clients  with  real 
instructional  design  problems  provide  opportunities  to  learn  through  immersion  in  the  culture  of  practice. 

Students  enrolled  in  the  IT  Clinic  are  expected  to  spend  considerable  time  on  course  activities  on  an 
ongoing  basis.  The  course  is  offered  for  variable  credit,  and  students  are  expected  to  spend  approximately  3 hours 
per  week  working  on  the  class  for  each  hour  of  course  credit.  Specifically,  students  are  expected  to: 

1 . Contribute  to  an  on-going  design  and  development  project. 

2.  Identify  project  and  individual  learning  needs. 

3.  Meet  project  goals  and  individual  learning  needs  in  a timely  manner. 

4.  Articulate  design  and  development  decisions. 

5.  Share  expertise  with  other  students. 

In  this  way,  the  IT  Clinical  supports  and  facilitates  situated  learning. 

Community  of  practice 

In  a community  of  practice,  participants  are  expected  to  relate  to  one  another  with  respect,  participate  in  and 
take  responsibility  for  the  collective  life  of  the  project,  and  count  on  one  another  for  meeting  both  individual  and 
collective  needs.  In  the  Clinical,  students  further  along  in  the  program  are  expected  to  collaborate  with  beginning 
students  and  mentor  them  along  with  faculty.  Individuals  are  ‘Tree  to  specialize  in  particular  areas  so  that  the 
community  can  capitalize  on  diversity”  (Lin,  et.  al.,  1995).  Clinical  students  assume  one  or  more  possible  roles, 
depending  on  the  project.  In  this  way,  students  learn  about  the  design  and  development  process  by  observing  how 
others  approach  a problem. 

Intrinsic  motivation 

Intrinsic  motivation  is  encouraged  as  students  set  personal  goals  to  perform  specific  activities  and  gain 
particular  skills.  Students’  progress  is  assessed  through  an  informal  system  in  which  students  determine  activities 
they  will  pursue  to  meet  those  goals  and  document  their  process.  As  faculty,  we  evaluate  the  reasonableness  of  those 
goals  and  the  extent  to  which  students  are  successful  in  meeting  them. 

Cooperation 

Students  work  together  to  accomplish  common  goals.  Each  participating  student  and  faculty  member 
evaluates  the  degree  to  which  their  peers  contributed  to  the  project  and  shared  their  expertise  with  others.  The 
willingness  of  participants  to  share  their  expertise  through  coaching  and  modeling  techniques  is  considered  in  these 
evaluations. 

Methods 

Methods  for  developing  expertise  include  modeling,  coaching,  scaffolding,  articulation,  reflection,  and 
exploration.  These  methods  are  clustered  into  three  groups  of  complementary  activities  (Collins,  et.  al.,  1991). 
Modeling , coaching , and  scaffolding  by  the  teacher  or  more  competent  peer  provide  learners  with  guidance  and 
support  as  they  gain  new  skills  through  progressively  more  complex  or  independent  activities.  Articulation  and 
reflection  by  both  the  teachers  and  students  make  reasoning  explicit  and  allows  learners  to  compare  their  developing 
understanding  with  the  ideas  of  others.  Exploration  provides  opportunities  for  learners  to  define  their  own  goal 
problems  and  seek  ways  of  solving  them  independently.  We  have  incorporated  each  of  these  features  in  our  Clinical 
experience. 

Modeling , coaching,  and  scaffolding 

Modeling  refers  to  the  demonstration  of  processes  and  strategies  to  perform  real-world  tasks  by  the  expert. 
In  a cognitive  apprenticeship,  many  problem  solving  tasks  take  place  mentally;  thus,  it  is  important  for  the  experts  to 
provide  detailed  explanations  of  what  they  are  doing  and  why.  Modeling  introduces  learners  to  the  processes  and 
strategies  used  to  solve  a complex  problem;  coaching  supports  students  as  they  attempt  a task  on  their  own. 

Coaching  involves  providing  hints,  feedback,  giving  encouragement,  and  otherwise  overseeing  the  learning  process 
(Collins,  et.  al.,  1991).  Scaffolding  refers  to  the  process  of  providing,  and  gradually  removing,  support  for  learners, 
as  they  become  increasingly  more  capable  of  performing  tasks  independently.  Support  can  range  from  doing  almost 
the  entire  task  for  the  learners  to  giving  occasional  hints  as  to  what  to  do  next.  Taken  together,  modeling,  coaching, 
and  scaffolding  provide  the  foundation  for  cognitive  apprenticeship  (Collins,  et.  al.,  1991) 


Our  Clinical  students  and  faculty  meet  once  a week  to  discuss  project  progress  and  plan  the  next  steps. 

Some  meetings  involve  the  client  as  well  as  the  project  team.  During  these  meetings,  we  scaffold  student  learning. 
Students  are  encouraged  to  perform  as  much  of  the  task  as  possible  independently.  We  constantly  assess  the  students 
performance,  and  when  their  performance  is  judged  to  be  going  astray  or  students  appear  to  be  floundering,  we 
coach  them  through  offering  suggestions,  providing  hints,  or  asking  guiding  questions.  If  they  still  appear  to  be 
going  astray  in  their  reasoning  or  performance,  we  step  in  and  model  the  way  we  would  handle  the  situation. 

These  weekly  design  meetings  provide  opportunities  for  faculty  to  diagnose  areas  where  the  students  are 
having  problems.  A variety  of  "just  in  time"  instruction  is  provided  to  demonstrate  skills  needed  by  the  design  and 
development  team  beyond  those  possessed  by  the  group  members.  Periodic  “workshops”  are  scheduled  throughout 
the  term  to  respond  to  the  learning  needs  of  a particular  semester.  Faculty  as  needed,  to  respond  to  the  information 
needs  of  the  task  at  hand  selects  Reading  materials.  Students  are  encouraged  to  learn  from  others  as  much  as 
possible.  In  fact,  students  are  expected  to  model  behaviors  for  their  peers,  coach  others  in  obtaining  new  skills,  and 
provide  scaffolding  for  their  peers. 

Reflection  and  Articulation 

Articulation  refers  to  the  process  of  verbalizing  thoughts  and  reasoning  in  order  to  make  design  decisions 
explicit.  Through  reflection,  individuals  compare  their  ideas  and  skills  with  those  of  others.  These  methods 
encourage  both  students  and  faculty  to  think  about  what  they  are  doing  and  to  make  their  tacit  knowledge  explicit. 
"By  engaging  students  and  faculty  in  dialogues  regarding  the  identification  of  problems  and  causal  relationships,  as 
well  as  requiring  justification  from  participants  regarding  their  use  of  certain  strategies,  cognitive  apprenticeships  are 
thought  to  encourage  reflective  thinking.  Reflection  is  believed  to  help  students  learn  how  to  think  about  their  work 
and  the  work  of  others  in  order  to  understand  it,  to  learn  from  it,  and  eventually  to  contribute  to  new  conceptions  of  it 
" (Ertmer  & Cennamo,  1995,  p.  48). 

During  weekly  Clinical  meetings,  faculty  and  students  reflect  on  and  articulate  their  design  decisions.  As  a 
means  of  assessing  student  learning,  students  are  expected  to  prepare  design  documents  (a  “case”)  based  on  their 
project.  Through  the  preparation  of  design  artifacts  and  cases,  students  are  required  to  reflect  on  their  current 
understanding  of  the  issues  of  the  project  and  identify  what  they  have  learned  by  participation  in  the  Clinical 
experience.  In  addition,  students  who  join  the  project  in  subsequent  semesters  are  able  to  examine  the  materials  in 
order  to  identify  the  current  status  of  the  design  project.  As  such,  the  cases  provide  an  orientation  to  the  project  for 
students  who  enroll  in  the  Clinical  in  subsequent  semesters. 

In  addition,  each  project  team  is  expected  to  publicly  present  their  project  in  forums  such  as  our 
Instructional  Technology  Professional  Seminar,  presentations  to  client  groups,  and  conference  presentations.  These 
presentations  provide  students  with  opportunities  to  reflect  on  their  decisions  and  articulate  their  reasoning  in  order 
to  justify  the  design  decisions  that  contributed  to  their  finished  product. 

Exploration 

" Exploration,  as  a method  of  teaching,  involves  setting  general  goals  for  students  and  then  encouraging 
them  to  focus  on  particular  subgoals  of  interest  to  them  or  even  to  revise  the  general  goal  as  they  come  upon 
something  more  interesting".  (Collins,  et.  al.,  1991,  p.  44)  Each  week,  Clinical  participants  identify  specific  tasks  to 
be  completed  before  the  next  class  meeting.  The  task  assigned  to  each  student  is  based  on  a combination  of  the 
students'  individual  strengths  as  well  as  their  interest  in  learning  more  about  unfamiliar  areas  of  instructional  design 
and/or  development.  Students  are  responsible  for  identifying  tasks  of  interest  to  them  and  setting  sub-goals  to 
accomplish  those  tasks.  Depending  on  the  nature  of  the  task,  students  may  work  individually,  in  pairs,  or  as  a group 
of  the  whole.  Students  generally  understand,  and  are  willing  to  accept  their  responsibilities  in  completing  an 
assigned  task 

Content 


Content  knowledge  necessary  for  expertise  includes  domain  knowledge,  heuristic  knowledge,  control 
strategies,  and  learning  strategies.  Although  students  are  expected  to  enter  our  course  with  domain  knowledge  in  the 
areas  of  instructional  design,  learning  and  instructional  theories,  and  technical  production  skills,  we  anticipate  that 
their  heuristic  strategies  and  control  strategies  may  be  relatively  undeveloped.  However,  as  the  students  have  at  least 
a bachelors  degree  before  enrolling  in  the  course,  we  assume  they  posses  adequate  knowledge  of  learning  strategies; 
thus  are  able  to  direct  and  monitor  their  own  learning.  Collins,  et.  al  (1991)  acknowledge  that  heuristic  knowledge 
and  control  strategies  may  be  learned  through  observation  of  experts  articulating  their  decisions,  making  their 
reasoning  explicit,  and  reflecting  on  what  they  did  and  why.  In  our  Clinical  classes,  faculty  and  students  work 
together  to  solve  instructional  problems.  The  composition  of  the  classes  reflect  a variety  of  expertise  in  instructional 
design  and  development,  ranging  from  instructors  with  fifteen  or  more  years  of  experience  in  the  field,  to  beginning 
masters  students  in  instructional  technology.  As  we  collaborate  as  a design  and  development  team,  we  try  to  develop 
heuristic  knowledge  and  control  strategies  through  the  sociology  and  methods  of  a cognitive  apprenticeship. 
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Sequencing 

Collins,  Brown  and  Holum  ( 1991 ) also  suggest  that  learning  activities  should  proceed  in  a certain  order. 
Global  activities  that  provide  opportunities  to  understand  the  whole  task  should  precede  the  development  of  specific 
skills.  Task  should  increase  in  complexity  and  difficulty,  and  finally,  activities  should  reflect  increasing  diversity  as 
students  develop  their  expertise  in  order  to  emphasize  the  broad  application  of  skills  across  a variety  of  situations. 
Although  this  sequencing  of  skill  development  is  not  apparent  within  a semester-long  Clinical  course,  students  are 
encouraged  to  enroll  in  the  Clinical  experience  for  multiple  semesters.  Likewise,  students  are  encouraged  to  enroll  in 
the  Clinical  at  multiple  points  in  their  academic  experience.  In  this  way,  beginning  students  may  develop  a global 
understanding  of  the  entire  design  and  development  experience  as  they  participate  in  less  complex  tasks  on  the 
development  team.  As  they  increase  their  expertise,  they  are  expected  to  assume  increasingly  complex 
responsibilities.  As  they  participate  in  the  course  across  multiple  semesters,  participating  in  multiple  projects,  they 
gain  practice  in  applying  skills  across  a variety  of  situations,  with  a variety  of  projects  and  clients. 

Procedures 

We  piloted  our  Clinical  idea  with  a group  of  5 students  during  the  spring  1998  semester.  During  this  pilot 
semester,  we  tested  and  refined  our  preliminary  ideas.  During  the  fall  1998  semester,  1 1 students  enrolled  in  two 
courses  offering  a Clinical  experience.  The  intent  of  this  paper  is  to  compare  students  expectations  of  the  Clinical 
experience  with  students  perceptions  of  the  actual  experience  (the  reality).  The  data  for  this  comparison  was 
collected  at  two  points  in  time. 

Data  Collection 
Expectations 

Data  reflecting  initial  expectations  of  the  Clinical  experience  was  collected  from  27  students  enrolled  in  our 
Professional  Development  seminar  during  the  fall  1997  semester.  Although  no  demographic  data  was  collected  from 
these  participants,  the  Professional  Development  seminar  is  intended  for  doctoral  students.  During  any  semester, 
typically  no  more  than  5 of  the  approximately  30  students  are  master  students.  Following  an  open  discussion  of  the 
Clinical  concept  of  approximately  30  minutes  in  length,  students  completed  an  open-ended  questionnaire  which 
asked  students  to  identify  the  benefits  of  such  an  experience,  their  related  concerns,  and  ideas  for  implementation. 
We  believed  that  this  advanced  group  of  students  would  be  in  an  ideal  position  to  reflect  upon  the  potential  benefits 
and  problems  of  a Clinical  experience.  This  data  reflects  students’  expectations  of  the  Clinical  experience. 

Reality 

During  the  fall  of  1998,  the  two  Clinical  courses  enrolled  six  master  students  and  four  doctoral  students 
majoring  in  Instructional  Technology.  One  doctoral  student  from  another  academic  area  was  enrolled  in  the  course. 
As  this  student  did  not  meet  the  characteristics  of  the  target  population  of  the  Clinical,  his  responses  were  eliminated 
from  the  final  analysis. 

Different  professors  taught  the  two  classes  and  each  class  worked  on  a different  project.  However,  both 
projects  were  complex  in  nature.  The  students  in  one  course  were  asked  to  create  an  online  system  to  support 
scientific  inquiry  using  museum  objects.  This  task  involved  providing  electronic  access  to  the  museum  collection 
and  developing  an  electronic  support  system  for  inquiry-based  learning.  Led  by  another  instructor,  students  in  the 
second  class  formed  a single  group  that  provided  services  for  a small  liberal  arts  university  in  North  Carolina.  This 
student  group  was  charged  with  the  responsibility  of  providing  "electronic"  solutions  to  instructional  problems 
identified  by  the  university's  faculty  and  staff. 

However,  the  intent  of  this  paper  is  not  to  discuss  the  details  of  the  individual  projects  but  to  discuss 
students'  perceptions  of  the  Clinical  experience  to  determine  commonalties  between  the  two  experiences.  In  the  fall 
1998  semester,  we  collected  quantitative  and  qualitative  data  on  the  perceptions  of  students  who  were  enrolled  in  the 
Clinical  courses.  At  the  time  at  which  the  data  was  collected,  these  students  had  participated  in  the  Clinical 
experience  for  fourteen  weeks.  This  data  represents  the  reality  of  the  Clinical  experience.  Students  were  asked  to 
rate  the  degree  to  which  they  agreed  or  disagreed  with  25  statements  derived  from  a list  of  possible  benefits  and 
concerns  generated  by  IT  Professional  Seminar  students  in  response  to  the  idea  of  having  Instructional  Technology 
students  work  on  "real  life"  projects  within  a course  setting.  They  rated  their  responses  on  a five-  point  scale  that 
included  "strongly  agree",  "agree”,  "don’t  know",  "disagree"  and  "strongly  disagree".  Statements  were  similar  to  the 
following: 

• I liked  being  pari  of  a team  working  toward  a common  goal. 

■ I benefited  from  interacting  with  clients  outside  of  IT. 

• I felt  that  the  faculty  treated  me  like  a colleague. 
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In  addition,  students  responded  to  three  open  ended  questions,  which  asked  them  to  reflect  on  the  situation, 
their  own  learning,  and  the  Clinical  experience.  They  submitted  written  responses  to  the  following  three  questions: 

1 . Assessment  of  situation  (For  example:  What  have  you  observed  when  working  on  this  project?  What  seemed  to  work  well? 
What  not  so  well?  What  would  you  have  liked  to  go  differently?  What  suggestions  would  you  have  for  others  who  wanted 
to  do  similar  projects?) 

2.  Self-  assessment  (For  example:  What  have  you  learned?  What  were  the  benefits  to  you?  Anything  you  wish  you  had  done 
differently?  What  caused  you  frustration?) 

3.  Assessment  of  course/  Clinical  (For  example:  What  were  the  benefits  of  the  way  it  was  conducted?  What  caused  you 
frustration?  What  suggestions  would  you  have  for  others  who  wanted  to  conduct  other  instructional  design  and 
development  Clinical  courses?) 

Results  and  Discussion 

Expectations 

Students'  expectations  were  analyzed  to  determine  trends  and  patterns  in  their  responses. 

Perceived  benefits  included  opportunities  for  peer,  faculty,  and  client  interactions;  professional  and  personal 
development;  opportunities  to  apply  what  they  learned  in  their  course  work,  and  available  resources.  They  were 
concerned  about  the  nature  of  peer,  faculty,  and  client  interactions,  scheduling  the  Clinical  within  the  current 
curriculum,  the  scope  of  the  task,  and  having  enough  resources. 

Peer  Interactions 

Students  perceived  increased  interactions  with  their  peers  to  be  a benefit  of  the  Clinical  experience.  They 
indicated  that  they  would  benefit  from  interaction  with  other  students  on  real  projects  and  by  being  a part  of  a team 
working  toward  common  goals.  On  the  other  hand,  they  were  concerned  about  trust  among  group  members. 

Faculty  Interactions 

They  viewed  interactions  with  faculty  members  as  both  a benefit  and  possible  concern.  They  perceived  that 
mentoring  and  modeling  of  behaviors  by  faculty  members  would  be  beneficial;  however,  they  were  concerned  that 
they  may  not  get  the  mentoring  they  needed.  They  realized  that  increased  interactions  with  IT  faculty  could  be 
beneficial,  yet,  they  were  concerned  that  faculty  may  not  be  actively  involved.  They  felt  it  would  be  beneficial  to  be 
treated  like  a colleague  by  the  faculty,  yet  were  concerned  that  they  may  not  be. 

Client  Interactions 

Students  also  viewed  client  interactions  as  beneficial  and  of  possible  concern.  They  thought  it  would  be 
beneficial  to  work  with  different  people  in  different  settings.  However,  they  were  concerned  that  they  may  not  have 
the  opportunity  to  be  a decision-maker. 

Task  Scope 

They  also  were  concerned  about  their  roles  and  responsibilities.  They  were  concerned  that  the  projects  may 
require  them  to  provide  technical  support  only,  and  hoped  to  have  the  opportunity  to  serve  as  instructional  designers 
as  well  as  developers.  In  addition,  they  were  concerned  that  the  task  may  be  too  broad  or  too  simple. 

Personal  and  Professional  Resources 

On  a related  note,  they  were  concerned  that  the  course  many  require  more  hours  of  work  than  a traditional 
course.  They  also  were  concerned  that  they  may  "get  in  over  their  head”  in  terms  of  technical  skills.  They 
recognized  that  "just  in  time"  training  was  a benefit  of  the  Clinical  experiences,  but  were  concerned  that  training  may 
not  be  provided  as  needed  or  that  they  would  not  be  given  the  time  to  learn  new  skills  when  needed.  They  felt  that 
access  to  on-campus  resources  might  be  a benefit,  yet  were  concerned  that  they  may  need  resources  that  were  not 
available. 

Professional  Development 

However,  they  saw  several  benefits  in  terms  of  professional  development.  They  perceived  that  the  real 
world  experience  would  provide  an  opportunity  to  apply  program  ideals  and  techniques.  They  saw  the  "hands-on" 
nature  of  the  course  as  extremely  practical.  Students  recognized  that  they  would  benefit  from  the  multiple 
perspectives  of  the  Clinical  participants.  They  believed  the  Clinical  experience  would  make  them  more  marketable, 
provide  satisfaction  for  the  work  done,  and  document  their  skills  and  competencies. 

Personal  Development 

They  also  looked  at  the  Clinical  experience  as  an  opportunity  to  determine  their  individual  strengths  and 
limitations.  They  believed  they  would  learn  to  delegate  responsibilities  and  accept  delegated  responsibilities. 

Students  also  thought  they  would  learn  to  respect  colleagues'  capabilities  and  communicate  with  them  about  specific 
goals. 


The  Reality 
Survey  Results 

Responses  to  the  survey  items  were  tabulated  and  percentages  of  students  selecting  each  response  option 
were  calculated.  For  most  of  the  survey  questions,  there  were  no  noticeable  differences  in  responses  between  the  two 
Clinical  courses.  Where  there  were  differences,  these  differences  are  noted.  Responses  are  discussed  in  relation  to 
students'  initial  expectations  in  each  category  of  perceived  benefits  and  concerns. 

Peer  Interactions 

Ninety  percent  of  the  students  felt  they  benefited  from  their  interactions  with  other  graduate  students.  All 
students  agreed  that  they  felt  like  part  of  a team.  Ninety  percent  of  the  students  liked  being  part  of  a team  working 
toward  a common  goal.  In  addition,  80%  felt  they  learned  skills  from  other  graduate  students  in  the  group.  The 
perceptions  of  students  who  had  completed  the  Clinical  experience  seemed  consistent  with  students'  expectations  of 
the  benefits  of  peer  interactions. 

Faculty  Interactions 

All  students  enjoyed  working  closely  with  IT  faculty  and  believed  that  the  faculty  was  actively  involved  in 
the  project.  Seventy  percent  of  the  students  felt  the  faculty  treated  them  like  colleagues.  The  other  30%  did  not 
know.  Ninety  percent  of  the  students  felt  they  benefited  from  the  mentoring  and  modeling  of  behaviors  by  faculty. 

Students’  perceptions  of  faculty  interactions  were  consistent  with  expected  benefits  of  the  experience. 
Students  initially  expressed  concern  that  the  faculty  may  not  be  actively  involved  in  the  project,  they  may  not  provide 
mentoring  as  needed,  and  that  they  may  not  treat  students  like  colleagues.  The  responses  of  students  who 
participated  in  the  Clinical  experience  indicates  these  concerns  were  unfounded. 

Client  Interaction 

Students  felt  they  benefited  from  interacting  with  clients  outside  of  the  field  of  Instructional  Technology. 
Eighty  percent  of  the  students  agreed  that  they  learned  to  communicate  with  clients  about  specific  goals.  However, 
two  students  from  one  class  disagreed  with  this  statement.  Students  expected  client  interactions  to  be  both  a benefit 
and  a concern;  from  the  responses  to  the  questionnaire,  client  interactions  seem  to  be  perceived  as  providing  more 
benefits  than  limitations 

Personal  and  Professional  Resources 

Students  initially  had  many  concerns  related  to  personal  and  professional  resources.  They  were  concerned 
that  the  course  many  require  more  hours  of  work  than  a traditional  course,  that  they  may  get  in  "over  their  head”  in 
terms  of  technical  skills,  that  training  may  not  be  provided  as  needed,  and  that  they  would  not  be  given  the  time  to 
leam  new  skills.  In  addition,  they  were  concerned  that  they  may  not  have  access  to  the  necessary  resources. 

Student's  responses  to  the  survey  indicated  that  these  concerns  were  unfounded. 

Seventy  percent  of  the  students  felt  they  received  training  and  instruction  as  needed.  In  one  of  the  classes, 
one  student  did  not  know  if  training  and  instruction  was  provided  as  needed,  and  two  students  believed  it  was  not 
provided  as  needed.  However,  all  students  felt  they  were  given  the  time  to  leam  skills  needed  on  the  projects;  and 
only  one  student  felt  "over  my  head"  in  terms  of  technology.  Eighty  percent  agreed  that  they  were  provided  with 
adequate  resources  to  complete  the  project,  one  student  disagreed,  and  one  didn’t  know.  Seventy  percent  of  the 
students  felt  the  time  demands  of  this  course  were  no  greater  than  for  a traditional  course;  half  of  the  students  in  one 
of  the  classes  (30%  of  respondents)  felt  the  time  demands  of  this  course  were  greater  than  in  traditional  courses. 

Students  were  split  in  their  feelings  toward  ending  their  involvement  after  one  semester.  Thirty  percent 
were  comfortable,  40%  didn't  know,  and  30%  were  not  comfortable  ending  their  involvement. 

Professional  Development 

Ninety  percent  of  the  students  felt  the  class  was  extremely  practical.  All  students  agreed  that  they  benefited 
from  the  different  perspectives  of  the  group  members,  faculty  and  clients.  Ninety  percent  of  the  students  agreed  that 
they  had  the  opportunity  to  apply  what  they  learned  in  other  classes.  The  survey  responses  confirm  students’  initial 
perceptions  of  the  professional  development  benefits  of  the  Clinical  experience. 

Personal  Development 

Ninety  percent  of  the  students  felt  they  learned  to  trust  their  colleagues.  Seventy  percent  of  the  students  felt 
they  learned  to  delegate  responsibilities.  However,  two  students  in  one  class  did  not  agree  that  they  learned  to 
delegate  responsibilities.  From  their  responses,  we  do  not  know  whether  they  already  knew  how  to  delegate 
responsibilities  or  whether  they  lack  the  skill  and  simply  did  not  leam  it.  However,  90%  agreed  that  they  learned  to 
accept  delegated  responsibilities.  All  students  agreed  that  they  learned  to  communicate  with  colleagues  about 
specific  goals.  Ninety  percent  agreed  that  they  learned  to  recognize  others  skills  and  capabilities.  Seventy  percent  of 
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the  students  agreed  that  they  became  aware  of  their  own  lack  of  abilities,  one  student  didn't  know,  one  disagreed  with 
the  statement,  and  one  did  not  respond.  From  the  responses  to  the  questionnaire,  it  appears  that  the  expected 
personal  development  opportunities  were  present. 

Results  of  Open-ended  Questions 

The  responses  to  the  open-ended  questions  were  analyzed  to  look  for  general  patterns  and  themes  across 
individuals.  Although  different  professors  taught  the  classes,  students  worked  on  different  projects,  and  course 
activities  differed  slightly,  analysis  of  the  qualitative  data  indicated  no  differences  in  students'  perceptions  of  the  two 
experiences. 

Student  comments  clustered  into  two  main  themes:  Students  primarily  commented  on  the  frustrations  of 
working  with  a client  and  the  benefits  of  the  experience.  In  the  discussion  below,  general  themes  are  illustrated  with 
student  comments. 

Client  Relations 

Even  though  the  two  courses  worked  on  different  projects  with  different  client  contacts,  the  majority  of 
comments  on  the  open-ended  questions  referred  to  client  relations.  Students  commented  on  their  difficulties  in 
obtaining  information  from  clients  as  quickly  as  they  would  like. 

■ "The  experiences  were  trying  at  times.  Clients  have  their  own  sense  of  time  and  time  frame.” 

■ "Communications  with  the  client  were  difficult  and  frustrating.  I learned  patience." 

■ "Timing  was  the  only  frustration.  Having  to  depend  on  someone  you  don't  even  know  for  information  is  difficult  to 
handle." 

Students  were  clearly  frustrated  with  the  difficulty  of  communicating  with  clients  about  their  goals, 
expected  products  and  vision. 

■ "I  think  the  client  caused  the  most  frustration.  I never  got  from  our  liaisons,  what  they  wanted,  nor  did  I really  understand 
the  view  they  were  coming  from." 

■ "In  hindsight,  it  appears  that  there  was  little  shared  understanding  of  the  issues  we  were  trying  to  grasp,  and  there  was  a 
long  drawn  out  period  where  these  were  slowly  teased  to  the  surface." 

■ "It  would  have  been  helpful  to  know  specifically  what  the  client  wanted  without  having  to  go  through  some  type  of 
discovery  process.  However,  this  would  be  counter  to  the  "real  world"  nature  of  the  course.” 

Students  wanted  to  share  what  they  learned  about  communicating  with  clients  with  others.  Advice  and 
suggestions  for  future  Clinical  groups  often  centered  on  client  relations. 

■ "I  would  suggest  to  others  to,  first  keep  it  professional,  don’t  just  listen  to  the  client,  hear  them,  and  hear  what  they  are  not 
saying,  and  don't  get  frustrated,  it  is  all  part  of  the  process." 

■ "..  . do  everything  within  your  power  to  coerce  a vision  out  of  your  client  before  proceeding,  even  if  it  means  extra  time  up 
front." 

■ "Make  sure  (hat  everyone’s  role  is  clearly  articulated  and  agreed  upon  before  proceeding.  . . . even  though  it  seems  overly 
formal  at  first,  it  can  avoid  problems  further  down  the  road." 

■ "Advice  for  such  situations  in  the  future:  make  sure  all  project  participants  are  clear  on  what  the  goals  and  products  of 
such  a partnership  will  be." 

Some  students  who  did  not  interact  with  the  client  directly  were  frustrated  by  this  lack  of  interaction. 

■ "I  have  enjoyed  being  in  the  Clinical.  I do  wish  I had  been  more  a part  of  the  client  process.  I feel  a lot  of  that  interaction 
was  done  by  others  and  at  times  I felt  on  the  outside  of  what  ever  was  going  on.  All  did  a great  job  of  reporting,  but  it  is 
hard  to  read  faces  and  gestures  from  others  repons." 

B "I  wished  that  I had  more  interaction  with  the  group.  I know  that  we  met  once  a week,  which  was  beneficial,  but  I feel  like 
I was  left  out  of  some  of  the  conversations  that  went  on." 

Benefits 


However,  students  enjoyed  the  experience  and  felt  they  benefited  from  participating.  Their  positive 
comments  reflect  our  success  in  developing  a supportive  apprenticeship  environment.  Students'  perceptions  will  be 
discussed  as  they  pertain  to  the  characteristics  of  a cognitive  apprenticeship  learning  environment. 

Situated  Learning 

Students  felt  their  learning  was  situated  in  a realistic  context  where  they  could  leam  skills  in  the  context  of 
their  application. 

■ "For  me  this  course  was  real  world.  The  task  was  handed  down,  assignments  were  made,  and  I was  responsible  for  doing 
my  part.  There  was  a team  of  people,  including  the  client,  depending  on  my  participation.  It  forced  me  not  to 
procrastinate  and  actively  take  a role  in  the  goals  and  objectives  of  the  program.  I would  gladly  take  more  classes  like  this, 
than  hand  holding  and  "this  is  what  could  happen"  kind  of  classes.  I enjoyed  the  experience  of  it  all." 

■ “I  feel  very  good  about  watching  and  participating  in  the  "real  process"  of  designing.  I think  this  experience  has  probably 
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laughi  me  more  than  if  we  had  had  a smooth  liner  process.  If  our  project  had  gone  according  to  the  books,  the  learning 
curve  for  me  would  have  been  much  lower." 


Community  of  Practice 

Students’  responses  to  the  open-ended  reflections  indicated  that  they  felt  they  were  part  of  a community  of 
practice,  where  each  team  member  was  working  toward  common  goals.  Students  commented: 

■ "The  benefits  of  the  Clinical  in  the  way  it  was  conducted  was  the  "real"  environment  that  was  created.  It  was  a work  group 
coming  together  and  delegating  task  to  one  another.  It  was  each  of  us  working  separately  toward  a common  goal  and 
needing  the  work  of  others  to  be  able  to  fit  our  part  into  the  puzzle." 

■ "I  appreciate  all  the  learning  I did  from  everyone.  Each  of  the  participants  brought  their  own  platter  to  the  table  and  I feel 
very  appreciative  that  I was  able  to  glean  some  knowledge  from  everyone." 

Intrinsic  motivation 

As  the  students'  comments  indicated,  they  felt  satisfaction  from  achieving  their  goals.  These  responses 
indicate  that  the  Clinical  successfully  created  a social  climate  that  encouraged  intrinsic  motivation. 


"It  is  an  exceptional  experience  to  see  work  done  for  class  to  have  an  immediate  impact  on  a client  or  institution  other  than 
doing  the  work  for  a grade  or  to  fulfill  a requirement." 

"This  class  was  a great  benefit  to  me.  I learned  what  it  takes  to  be  a team  to  achieve  a common  goal.  What  I really  liked 
about  this  class  was  that  it  pertained  to  a "real"  situation.  It  felt  good  once  the  goal  was  achieved." 


Collaboration 

Students  also  felt  they  worked  in  a collaborative  environment,  where  they  were  part  of  a team. 


"That  was  the  best  thing  about  the  experience...  for  everything  that  we  needed  to  get  done  there  was  one  of  us  that  could 
do  it  without  much  difficulty." 

"I  also  learned  the  importance  of  doing  what  you  do  well  whenever  you  can.  It  made  it  a lot  easier  to  get  the  easy  stuff 
done,  let  other  people  gel  their  easy  stuff  done,  and  hopefully,  there  will  be  nothing  left." 


Students  also  recognized  the  value  of  the  methods  of  instruction.  They  commented  on  the  benefits  of  the 
coaching,  modeling,  and  scaffolding  that  occurred. 

■ “The  Clinical  experience  was  good....  I really  grew  from  the  mentoring  and  discussions." 

■ "I  learned  a great  deal  from  all  of  the  people  in  the  group.  Working  with  others  has  given  me  a better  understanding  of  the 
process  and  what  to  do  and  what  not  to  do." 


Student  responses  indicated  active  articulation  and  reflection  on  their  skill  development. 

■ "The  benefits  to  me  were  the  group  interaction  and  being  responsible  not  only  for  myself  but  also  for  a team  of  people,  the 
interaction  with  the  client  and  feeling  like  my  skills  were  ‘up  there’  and  'comparable  to  others'." 


And  finally,  they  rose  to  the  challenge  inherent  in  the  exploration  phase  of  a cognitive  apprenticeship. 

■ “I  had  to  learn  new  skills  as  I applied  them.  1 would  have  preferred  to  have  the  skill  going  into  the  class.  However,  this  is 
often  the  reality  of  the  work  place.” 


Conclusions 

The  results  of  our  investigation  indicate  that  we  successfully  established  an  environment  conducive  to 
apprenticeship  learning  in  our  Instructional  Technology  Clinical  courses.  We  compared  students'  expectations  of  the 
Clinical  experience  in  terms  of  perceived  benefits  and  concerns  with  students  perceptions  of  the  reality.  The  results 
indicated  that  the  expected  benefits  from  faculty  and  peer  interactions,  realistic  experiences,  client  interactions, 
personal  and  professional  development  opportunities,  and  applied  learning  were  achieved.  In  addition,  the  results 
indicated  that  students'  concerns  about  inadequate  training  and  mentoring  by  faculty,  increased  time  demands  above 
those  of  traditional  courses,  lack  of  trust  among  team  members,  and  inadequate  resources  were  unjustified.  Analysis 
of  students’  reflections  on  the  Clinical  courses  indicates  that  students  perceived  the  sociology  and  methods  of 
cognitive  apprenticeships  as  beneficial  to  the  Clinical  experience.  Their  responses  mentioned  the  beneficial  effects 
of  modeling,  coaching,  scaffolding,  reflection,  articulation,  exploration,  situated  learning,  cooperation,  intrinsic 
motivation,  and  a community  of  practice. 

However,  Clinical  students  expressed  frustrations  at  the  nature  of  their  interactions  with  clients.  It  must  be 
noted  that  from  our  perspective  as  faculty  members  with  extensive  experience  in  design  and  development,  our 
interactions  with  the  clients  were  not  atypical.  In  many  ways,  the  students'  concerns  were  justified:  their  concerns 
were  the  result  of  the  situated  nature  of  their  learning.  They  were  frustrated  as  they  tried  to  apply  skills  in  client 
interactions;  skills  that  are  not  addressed  by  typical  coursework  in  instructional  technology,  but  are  best  developed 
through  experience  and  immersion  in  the  culture  of  practice.  The  prevalence  of  student  comments  on  client  relations 
emphasizes  the  need  for  experiential  learning  experiences  in  instructional  technology  graduate  programs.  By 
participating  in  real  experiences  with  real  clients  prior  to  graduation,  students  can  be  mentored  and  discuss  possible 
strategies  with  colleagues  in  a safe  learning  environment. 

The  results  of  these  analyses  suggest  that  we  may  need  to  prepare  both  clients  and  students  for  the 
collaborative  experiences.  Both  clients  and  students  need  to  realize  that  the  client’s  environment  and  university 
classes  represent  different  cultures,  each  with  their  own  set  of  expectations,  vocabulary,  timelines,  rewards,  and 
consequences.  As  faculty  coordinators,  we  should  assess  students’  experience  in  interacting  with  clients  to  design 
and  develop  instructional  materials  and  conversely,  clients’  experience  in  working  with  instructional  designers  and 
developers.  Guidance  can  be  provided  to  both  clients  and  students  in  the  form  of  readings,  direct  modeling, 
coaching  behaviors,  and  discussions  concerning  issues  of  collaborating  on  a common  project. 
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Despite  occasional  frustrations  by  students,  both  groups  gained  from  their  collaboration.  The  students 
working  with  the  university  in  North  Carolina  successfully  accomplished  the  following  tasks: 

■ wrote  and  submitted  a grant  to  request  funding  for  the  establishment  and  operation  of  an  instructional  systems 
development  research  group. 

■ designed  and  developed  an  on-line  database  to  assist  the  university  in  procurement,  security,  and  maintenance  of 
instructional  technology  facilities  and  equipment 

■ negotiated  an  agreement  to  enroll  participants  of  the  university's  cohort  initiative  into  a nationally  acclaimed  professional 
development  academy; 

■ reviewed  and  reported  on  a variety  of  software  tools  used  to  develop  and  deliver  Web-based  instruction 

■ arranged  for  the  cohort's  graphic-design  work  to  be  sub-contracted  to  an  external  group  comprised  of  work-study  students 
majoring  in  graphic  arts. 

Students  working  with  the  museum  successfully: 

■ reviewed  literature  on  inquiry  based  learning 

■ reviewed  numerous  museum  web  sites  to  identify  those  that  included  elements  of  inquiry  based  learning 

■ developed  a plan  for  a museum  web  site  that  provided  access  to  collection  information  for  the  purposes  of  conducting 
inquiry 

■ developed  a prototype  web  site  that  included  collection  images  and  data,  inquiry  based  iesson  plans,  and  background 
information  on  scientific  inquiry. 

We  envision  this  Clinical  in  product  design  and  development  as  the  first  of  several  such  experiences.  Our 
graduates,  like  our  faculty,  often  have  responsibility  for  design  and  development  of  instructional  materials,  for  the 
creation  and  publication  of  original  research,  and  for  teaching.  Apprenticeship  learning  experiences  can  be 
developed  in  both  research  and  teaching  based  on  this  model. 
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Abstract 

The  purpose  of  several  studies  was  to  examine  the  effects  that  differentially  coded  (black  and  white  and 
color)  illustrations  had  on  students  who  were  classified  as  field  independent , field  neutral  and  field 
dependent  on  tests  measuring  different  educational  objectives.  The  effects  of  visual  and  verbal  formats 
were  also  examined.  In  general,  results  reveal  an  insignificant  interaction  between  coding  type  and  level 
of  field  dependence.  However,  field  dependency  was  found  to  be  an  important  instructional  variable , and 
that  for  some  types  of  learning  objectives  the  process  of  color  coding  instructional  materials  may  reduce 
achievement  differences  attributed  to  differences  in  cognitive  style.  Unexpectedly,  students  who  received 
verbal  tests  across  all  field  dependence  levels  achieved  significant  higher  mean  scores  than  did  those  who 
received  the  visual  test  formats. 

With  the  response  capabilities  inherent  in  the  new  electronic  technologies,  educators  are  becoming 
increasingly  interested  in  examining  instructional  effects  of  the  learners’  cognitive  learning  style.  Cognitive  learning 
style  is  generally  considered  to  describe  the  manner  in  which  an  individual  interacts  with  and  processes  information. 
Probably  the  most  extensively  researched  cognitive  style  which  has  the  widest  application  to  educational  problems  is 
known  as  field  dependence/independence  (Witkin,  Dyk,  Faterson,  Goodenough,  & Karp,  1962).  The  field 
dependence/independence  continuum,  as  it  applies  to  learners,  describes  the  degree  to  which  learners  will  interact 
with  a visual  presentation,  that  is,  whether  the  learner  will  merely  interact  with  the  visual  as  presented  or  will  he/she 
analyze,  reorganize  and  synthesize  the  instructional  field  to  make  the  content  more  meaningful  and  memorable 
(Ausbum,  & Ausbum,  1978).  Field  dependent  individuals,  when  presented  a visualized  presentation  tend  to  modify 
the  structure  but  accept  and  interact  with  it  as  it  is  presented.  They  tend  to  fuse  all  segments  within  the  visual  field 
and  do  not  view  or  interact  with  the  visual  components  discretely.  Field  independents  tend  to  act  upon  a visual 
stimulus,  analyzing  it  when  it  is  organized  and  providing  their  own  structure  when  it  lacks  organization.  The 
findings  of  most  studies  in  the  literature  examining  achievement  indicate  a superiority  of  FID  students  over  FD 
student  in  terms  of  performance  (Moore,  1985;  Reardon,  & Moore,  1988;  Canelos,  Taylor,  & Gates,  1980;  Witkin, 
Moore,  Goodenough,  & Cox,  1977).  These  results  are  explained  by  the  fact  that  FID  student  utilize  their 
competence  in  articulation  or  in  analyses  and  structuring  (Couch,  & Moore,  1992). 

Although  many  studies  have  examined  the  effects  of  visual  attributes  on  learning  (Dwyer,  1978;  Dwyer, 
1987)  few  have  studied  the  effects  of  varied  visual  attributes  on  specific  cognitive  learning  styles.  Research  has 
shown  that  color  coding  helps  learners  organize  or  categorize  information  into  useful  patterns  which  enables  them  to 
interpret  and  adjust  more  readily  to  their  environment.  It  was  hypothesized  that  color-coded  visuals  would  be  more 
effective  than  black  and  white -coded  visuals  in  enhancing  the  solvent  visual  cues,  thereby  making  them  more 
identifiable  and  instructional  to  field  dependent  learners.  The  color  coding  would  attempt  to  compensate  for  the 
restructuring  skills  absent  in  the  field  dependent  learners  and  subsequently  lead  to  deeper  information  processing  and 
increased  achievement.  This  hypothesis  seemed  plausible  since  field  dependent  learners  tend  to  be  global  in 
perception  and  would  be  most  inclined  to  take  advantage  of  the  increased  structure  provided  by  the  color  coding. 

Previous  research  (Dwyer,  1978;  Dwyer,  1987;  Dwyer,  1968)  has  verified  the  superiority  of  color  coded 
visuals  over  black  and  white  illustrations  in  answering  questions  of  a verbal  nature,  i.e.,  terminology  and 
comprehension  (Dwyer,  1978;  Dwyer,  1987).  However,  there  was  no  research  available  which  investigated  the  role 
of  color  on  verbal  or  visual  type  questions  across  the  cognitive  style  known  as  field  dependence/independence 
(FD/I).  Additionally,  it  was  not  known  whether  coding  (B&W  and  Color)  and  field  dependence  interact  or  whether 
color  coding  positively  affects  the  deficiency  of  field  dependent  learners  to  organize  and  structure  instructional 
content. 

Statement  of  the  Problem 

Specifically,  the  purpose  of  these  studies  was  to  examine  the  effect  that  coding  (B&W  and  color)  has  on  the 
achievement  of  students  categorized  as  FD/I  learners  and  to  determine  if  there  was  an  interaction  between  field 
dependency  and  color  across  both  visual  and  verbal  oriented  tests  measuring  different  educational  objectives. 
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Instructional  Treatments 


The  subject  content  for  the  studies  consisted  of  a 2,000  word  instructional  booklet  on  the  anatomy  and 
functions  of  the  human  heart.  Each  booklet  contained  nineteen  illustrations  which  were  designed  to  illustrate  the 
content  being  presented  verbally.  Booklets  received  by  the  students  in  the  black  and  white  and  color  coded 
treatments  were  identical  in  verbal  and  visual  content  with  the  exception  of  the  color  code  which  was  applied  to  one 
treatment. 

The  instructional  material  consisted  of  a 7”  by  8-1/2”  spiral  bound  booklet.  The  booklet  included  one  page 
of  directions  and  twenty  pages  of  concepts  and  functions  on  the  heart  integrated  by  prose  text  with  accompanying 
visualization.  A simple  line  drawing  was  positioned  (2-3/4”  by  3-1/8”)  on  each  frame.  All  booklet  pages  were 
clearly  numbered,  with  the  prose  text  in  the  lower  half  of  each  page  and  the  visualization  in  the  upper  half  of  the 
page.  The  arrows  were  employed  to  focus  learner  attention  on  relevant  information  in  the  visual. 

Directions  on  the  first  page  of  the  booklet  indicated  to  the  learner  that  the  materials  were  part  of  an 
investigation  to  study  the  relative  effectiveness  of  visual  illustrations  which  accompanied  printed  instruction. 
Learners  were  directed  to  be  attentive  to  both  the  written  and  visual  information.  They  were  further  instructed  that 
there  was  no  time  limit  to  the  instructional  sequence,  and  that  a test  with  several  parts  would  be  administered  when 
they  felt  they  had  mastered  the  presented  concepts. 

Given  all  of  the  above  design  elements  found  within  the  black/white  instructional  booklets,  the  color  coded 
treatment  differed  only  in  the  application  of  color  to  certain  verbal  and  visual  concepts  illustrates  the  color  coding 
application.  Six  color  categories  and  black  were  used  to  code  the  heart  concepts. 

To  facilitate  replication  printer’s  ink  colors  selected  were  from  the  Pantone  Matching  System,  developed  by 
Pantone  Incorporated.  The  six  colors  were:  red  (Pantone— warm  red),  blue  (Pantone-process  blue),  green  (Pantone- 
-green),  purple  (Pantone— No. 227),  brown  (Pantone-No.471),  and  gold  (Pantone-No.  124).  The  black  (Pantone— 
black)  was  used  for  all  the  non-color  coded  design  elements  within  the  color  coded  material  and  for  all  the  design 
elements  within  the  black/white  material.  The  illustrations  in  the  black  and  white  version  contained  black  and  white 
coded  line  drawings  which  highlighted  the  information  and  process  being  presented.  Students  receiving  the  color 
coded  treatment  received  the  same  visuals  as  did  students  receiving  the  black  and  white  treatment;  however, .several 
different  colors  were  used  to  highlight  the  information  being  discussed.  The  major  independent  variables  in  the 
studies  were  the  effect  that  B&W  and  Color  Coding  of  information  had  on  the  information  processing  strategies  of 
students  identified  as  processing  different  levels  of  field  dependence  (FI,  FN,  FD). 

Criterion  Measures 

Each  student  in  each  treatment  received  the  Group  Embedded  Figures  Test  (GEFT)  (Witkin,  Oltman, 
Raskin,  & Karp,  1971).  The  GEFT  (1 1)  is  a group-administered,  25  item  test  administered  in  three  timed  sections 
(2,2  and  5 minutes  each).  Students  must  trace  one  of  eight  simple  figures  embedded  in  figures  of  greater  complexity. 
Reported  reliability  is  .82.  Students  participating  in  these  studies  were  classified  as  field  independent  (FI),  field 
neutral  (FN)  in  field  dependent  (PD)  based  on  their  performance  on  the  GEFT.  The  grand  mean  and  standard 
deviations  was  calculated  for  each  study.  Students  achieving  one-half  standard  deviation  above  the  grand  mean  were 
considered  to  be  FI  while  students  achieving  one-half  standard  below  the  mean  were  considered  to  be  FD.  Field 
neutral  students  were  those  achieving  one-half  standard  deviation  on  either  side  of  the  mean. 

After  receiving  their  respective  coded  treatments  each  student  then  received  a battery  of  individual  tests  in 
either  a verbal  or  visual  format.  These  tests  were  designed  to  measure  achievement  of  different  types  of  educational 
objectives.  The  following  description  of  the  criterion  tests,  adapted  from  Dwyer  (1978,  pp.45-47)  illustrated  the 
types  of  instructional  objectives  assessed  in  this  study. 

Drawing  Test 

The  objective  of  the  drawing  test  was  to  evaluate  student  ability  to  construct  and/or  reproduce  items  in  their 
appropriate  context.  The  drawing  test  provided  the  students  with  a numbered  list  of  terms  corresponding  to  the  parts 
of  the  heart  discussed  in  the  instructional  presentation.  The  students  were  required  to  draw  a representative  diagram 
of  the  heart  and  place  the  numbers  of  the  listed  parts  in  their  respective  positions.  For  this  test  the  emphasis  was  on 
the  correct  positioning  of  the  verbal  symbols  with  respect  to  one  another  and  in  respect  to  their  concrete  referents. 
Identification  Test 

The  objective  of  the  identification  test  was  to  evaluate  student  ability  to  identify  parts  or  positions  of  an 
object.  This  multiple-choice  test  required  students  to  identify  the  numbered  parts  on  a detailed  drawing  of  a heart. 
Each  part  of  the  heart  which  had  been  discussed  in  the  presentation  was  numbered  on  a drawing.  The  objective  of 
this  test  was  to  measure  the  ability  of  the  student  to  use  visual  cues  to  discriminate  one  structure  of  the  heart  from 
another  and  to  associate  specific  parts  of  the  heart  with  their  proper  names. 
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Terminology  Test.  This  test  consisted  of  items  designed  to  measure  knowledge  of  specific  facts,  terms, 
and  definitions.  The  objectives  measured  by  this  type  of  test  are  appropriate  to  all  content  areas  which  have  an 
understanding  of  the  basic  elements  as  a prerequisite  to  the  learning  of  concepts,  rules,  and  principles. 

Comprehension  Test  Given  the  location  of  certain  parts  of  the  heart  at  a particular  moment  of  its 
functioning,  the  student  was  asked  to  determine  the  position  of  other  specified  parts  or  positions  of  other  specified 
parts  of  the  heart  at  the  same  time.  This  test  required  that  the  students  have  a thorough  understanding  of  the  heart,  its 
parts,  its  internal  functioning,  and  the  simultaneous  processes  occurring  during  the  systolic  and  diastolic  phases.  The 
comprehension  test  was  designed  to  measure  a type  of  understanding  in  which  the  individual  can  use  the  information 
being  received  to  explain  some  other  phenomenon. 

Total  Test  Score.  The  items  contained  in  the  individual  criterion  tests  were  combined  into  a composite  test 
score.  The  purpose  was  to  measure  total  achievement  of  the  objectives  presented  in  the  instructional  unit. 

The  design  of  the  visual  form  of  each  of  the  criterion  tests  utilized  only  one  drawing  with  four  or  five  letter 
labels  in  all  items  in  which  it  was  possible  to  do  so  while  maintaining  clarity  and  correspondence  to  the  verbal  test 
items.  However,  two  items  in  the  terminology  test  and  all  items  in  the  comprehension  test  required  four  drawings. 
The  item  stems  of  both  the  verbal  and  visual  test  questions  were  verbal  and  asked  the  same  question.  In  addition,  the 
visual  distracters  in  the  visual  tests  corresponded  to  the  verbal  distracters  in  the  verbal  tests  as  closely  as  was 
reasonable.  The  description  of  the  verbal  tests  given  previously  also  describes  the  visual  tests. 

Summaries  of  Studies  Investigating  Color  Coding  and  Cognitive  Style 

Study  1.  Moore,  D.M.  & Dwyer,  F.M.  (1991).  Effect  of  color  coded  information  on  students’  level  of  field  dependence.  Perceptual  and  Motor 
Skills.  72,611-616. 

Number  of  subjects:  N=1 19 

Independent  variables:  black  and  white  and  color  coded  treatments;  levels  of  field  dependence  (Fi/FN/FD). 

Dependent  variables:  terminology,  comprehension  and  total  test  (combined  terminology  and  comprehension). 

Number  of  Students  in  each  level  of  field  dependence:  Fl=43,  FN=45,  FD=29. 

Results:  Insignificant  interaction  between  coding  strategy,  (black  and  white  and  color  coded)  and  levels  of  field  dependence  (FI, 

FN,  FD). 


Field  independent  students  achieved  significantly  higher  mean  scores  on  both  the  terminology  and  comprehension  tests 
when  receiving  the  black  and  white  coded  treatments;  however,  when  color  coded  treatments  were  used  insignificant  differences  in  achievement 
resulted  between  the  FI  and  FD  students. 

On  the  terminology  and  comprehension  tests  insignificant  differences  in  mean  achievement  resulted  between  students 
receiving  the  black  and  white  and  color  coded  treatments. 


Study  2.  Dwyer,  F.M.  & Moore,  D.M.  (1991).  Effect  of  Color  Coding  on  visualty  amended  tests  with  students  of  different  cognitive  styles. 
The  Journal  of  Psychology.  125,  677-680. 


Number  of  subjects:  N=  1 1 7 

Independent  variables:  black  and  white  and  color  coded  treatments;  levels  of  Field  dependence  (FI,  FN,  FD). 

Dependent  variables:  scores  on  the  two  visual  oriented  tests-drawing  and  identification.  Total  test  was  a combination  of  both 
criterion  measures. 

Number  of  Students  in  each  level  of  field  dependence:  FI=43,  FN=29,  FD=45. 

Results:  Across  all  levels  of  cognitive  (FI,  FN,  FD)  significant  differences  were  found  to  exist  for  cognitive  style  and  color  coding 

on  the  drawing,  identification  and  total  tests.  Students  at  all  levels  who  received  the  color  coded  treatment  achieved  significantly  higher  scores 
than  did  students  who  received  the  black  and  white  color  coded  treatments. 

On  the  drawing  test  FI  students  scored  significantly  higher  than  did  FD  students  on  both  the  black  and  white  and  color  coded 
treatments.  On  the  identification  test  achievement  differences  between  students  receiving  the  black  and  white  and  color  coded  treatments  were 
insignificant. 

Study  3.  Dwyer,  F.M.  & Moore,  D.M.  (1991-92).  Effect  of  color  coding  on  visually  and  verbally  oriented  tests  with  students  of  different  field 
independence  levels.  Journal  of  Educational  Technology  Systems.  20,  31 1-20. 

Number  of  subjects:  N=  119 

Independent  variables:  black  and  white  and  color  coded  treatments,  level  of  field  dependence  (FI,  FN, 

FD). 


Dependent  variables:  achievement  scores  on  the  drawing,  identification,  terminology,  comprehension  and  total  criterion  test 

(combination  of  the  instructional  criterion  measures 

Number  of  Students  in  each  level  of  field  dependence:  FI=43,FN=45,FD=29: 

Results:  An  insignificant  interaction  was  found  to  exist  between  coding  strategy  and  levels  of  field  dependence.  On  all  criterion 
measures  for  all  levels  of  field  dependence,  students  who  received  the  color  coded  treatments  achieved  significantly  higher  scores  than  did 
students  receiving  the  black  and  white  coded  treatment. 

In  analyzing  individual  performance  on  the  drawing,  identification  and  total  criterion  test,  students  receiving  the  color  coded 
treatments  achieved  significantly  higher  mean  scores  than  did  students  receiving  the  black  and  white  coded  treatments.  However,  on  the 
terminology  and  comprehension  tests  there  were  insignificant  differences  in  achievement  between  students  receiving  the  black  and  white  and 
color  coded  treatments. 

Significant  differences  existed  in  favor  of  FI  students  on  all  criterion  measures  when  they  received  the  black  and  white  coded 
treatments;  however,  when  students  received  the  color  coded  treatments  insignificant  differences  were  found  to  exist  on  the  four 
individual  criterion  measures. 

Study  4.  Moore,  D.M.  & Dwyer,  F.M.  (1994).  Effect  of  cognitive  style  in  test  type  (visual  or  verbal)  and  color  codings  Perceptual  and  Motor 
Skills.  79.  1532-1534. 

Number  of  subjects:  N=183 

Independent  variables:  black  and  white  and  color  coded  treatments;  levels  of  field  dependence(FI,  FN,  FD),  visual  and  verbal  test 

formats. 

Dependent  variables:  scores  on  both  the  verbal  and  visual  test  formats  for  the  terminology,  identification,  drawing  comprehension 
and  total  criterion  measures. 

Number  of  Students  in  each  level  of  field  dependence:  FI=66,  FN=74,  FD=43. 

Results:  On  all  criterion  measures  significant  achievement  differences  occurred  in  favor  of  FI  students  on  color  coding  in  favor  of 

color  coding.  Students  receiving  the  verbal  test  formats  achieved  significantly  higher  achievement  scores  on  all  criterion  measures  than  did 
students  receiving  the  visual  test  formats. 

Results  and  Discussion 


The  results  of  these  experimental  studies  support  the  contention  that  field  independent  and  field  dependent 
learners  differ  in  the  cognitive  processes  they  use  as  in  the  effectiveness  of  these  cognitive  processes  as  measured  on 
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tests  measuring  different  educational  objectives.  The  studies  found  that  across  all  studies  where  FI  and  FD  students 
reviewed  black  and  white  coded  treatments,  the  field  independent  students  scored  significantly  higher  than  did  field 
dependent  students  on  the  terminology/drawing,  terminology/comprehension  and  total  criterion  tests.  These  results 
support  prior  research  (Moore,  & Bedient,  1986)  which  found  that  independent  learners  tend  to  score  higher  on 
criterion  measures  which  require  the  acquisition  of  information  from  visualized  instruction  and  are  used  to  assessing 
visually  complemented  instruction.  This  finding  is  also  consistent  with  the  previous  reviews  of  the  literature  that 
have  concluded  that  field  independent  learners  exhibit  an  active,  hypothesis  testing  strategy  toward  learning,  whereas 
field  dependent  learners  tend  to  employ  a more  tentative  or  spectator  approval  to  learning  (Witkin,  Moore, 
Goodenough,  & Cox,  1977).  The  results  also  indicated  that  across  all  studies  on  a number  of  criterion  measures 
(e.g.,  drawing  and  identification  tests)  the  use  of  color  coding  of  visualization  did  not  provide  sufficient  structuring 
of  the  critical  information  to  alter  the  information  processing  level  of  field  dependent  learners. 

It  was  expected  that  the  color  coded  treatments  would  make  the  relevant  cues  more  explicit  to  the  field 
dependent  learners,  thereby  improving  their  performance.  When  both  FI  and  FD  students  both  received  the  color 
coded  treatments  insignificant  differences  in  mean  achievement  on  all  four  individual  criterion  measures. 

Apparently,  the  color  coded  illustrations  provided  a sufficient  structure  for  the  FD  learners  to  interact  with  and 
internalize  at  levels  similar  to  that  achieved  by  the  FI  learners.  Another  possible  explanation  may  be  presented  to 
explain  these  results.  Witkin,  Moore,  Goodenough,  and  Cox  (i977).  have  indicated  that  it  is  possible  to  induce  FD 
learners  to  use  analytical  techniques  by  providing  specific  directions  as  to  how  to  proceed.  Possibly  the  specific 
directions  on  the  first  page  of  the  instructional  treatments  was  sufficiently  strong  to  instigate  greater  levels  of 
information  processing  thereby  enabling  them  to  achieve  at  levels  realized  by  the  FD  learners.  Annis*  study  (1979) 
investigating  cognitive  style  on  study  technique  found  that  field  dependent  students  did  not  score  as  well  as  field 
independent  students  in  completing  items  of  high  structural  importance  even  when  the  passage  was  well  organized. 
The  implication  being  that  the  field  dependent  students,  in  addition  to  receiving  the  well  organized  instructional 
module  received  specific  directions  alerting  them  to  the  fact  that  would  be  tested  upon  completion  of  the  module. 

The  combination  of  specific  directions  and  the  color  coded  illustrations  enabled  the  field  dependent  students  to 
achieve  levels  similar  to  those  achieved  by  the  field  independent  students. 

It  was  hypothesized  that  the  students  who  received  the  visual  version  of  the  criterion  measures  would 
achieve  significant  higher  scores  in  the  criterion  measures.  This  was  not  the  case,  in  fact,  students  who  received  the 
verbal  version  of  the  criterion  measures  achieved  significantly  higher  scores  than  did  students  who  received  the 
visual  versions.  These  results  were  unexpected  and  require  further  study  before  definitive  conclusions  can  be 
proposed. 

Note:  Presentation  based  upon  an  article  entitled:  Dwyer,  F.M.,  & Moore,  D.M.  (1997-1998).  Field  dependence  and 
color  coding;  A review  and  summary  of  research  evidence.  Journal  of  Educational  Technology  Systems.  26  (3), 
243-253. 
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Abstract 

Three  hundred  learners  participated  in  this  study  which  (a)  examined  the  instructional  effectiveness  with 
which  different  job  aid  types  facilitated  learner  achievement  of  different  types  of  learning  objectives  and 
(b)  identified  the  degree  to  which  low  and  high  prior  knowledge  individuals  profit  from  different  job  aid 
types.  The  instructional  content  used  in  the  study  was  an  instructional  module  on  the  parts  and  functions  of 
the  human  heart.  Participants  were  randomly  assigned  to  one  of  five  treatments  complemented  with 
different  job  aid  types.  After  interacting  with  the  instructional  module ; participants  received  four  post  tests 
each  measuring  different  educational  objectives.  Research  findings  indicated  that  when  job  aids  are  used 
following  instruction  (a)  they  are  not  equally  effective  in  facilitating  learner  achievement  of  different  types 
of  educational  objectives;  (b)  they  do  not  reduce  learning  differences  between  low  and  high  prior 
knowledge  learners ; and  (c)  there  is  an  insignificant  interaction  between  job  aid  type  and  prior  knowledge 
level. 

Introduction 

The  essence  of  appropriate  instruction  relies  heavily  on  an  instructional  design  practitioner’s  ability  to 
balance  instructional  efficiency  and  effectiveness.  Instructional  efficiency  takes  into  account  finite  resources  such  as 
time,  money,  and  staff  personnel  allocated  to  the  development  of  an  instructional  intervention  while  instructional 
effectiveness  focuses  on  the  desired  performance  or  outcome  of  that  intervention.  To  compound  this  situation  in  the 
corporate  environment,  performance  demands  change  constantly  because  of  the  introduction  of  new  products, 
services,  process  and  technology;  individuals  are  required  to  maintain  current  performance  levels  while  acquiring 
new  complex  processes  and  skills;  and  management’s  expectations  have  employees  performing  more  tasks  at 
irregular  intervals.  In  meeting  some  of  these  diverse  demands,  instructional  design  practitioners  have  relied  on  non- 
instructional  interventions  known  as  job  aids  (Silber,  1990).  Job  aids,  also  known  as  performance  support  tools, 
contain  factual  and  procedural  knowledge  and  are  used  during  actual  task  completion  (Grau,  1986;  Rossett,  1991; 
Rothwell  & Kazanas,  1994).  They  are  effective  especially  when  the  consequence  of  error  is  high,  performance  is 
lengthy  or  complex,  performance  is  changing  frequently  or  limited  budget  or  time  exists  for  making  an  instructional 
intervention  (Carlisle  & Coulter,  1990;  Finnegan,  1985). 

Duncan  (1985,  p.l)  indicates  that  job  aids  “...put  more  training  into  the  programs  of  instruction  without 
significantly  increasing  the  course  length,  ...save  time  and  money  in  training  development  without  sacrificing  student 
achievement,  ...reduce  the  paperwork  requirements  in  training  development  and  multitude  of  training 
products... and... increase  performance  both  initially  and  on  a sustained  basis.”  Finnegan  (1985)  proposes  the 
following  five  reasons  to  use  any  job  aid  type:  (1)  to  provide  a performance  focus,  (2)  to  guide  performance  that 
would  likely  be  forgotten,  (3)  to  reduce  cost  and  development  time — as  compared  to  training,  (4)  to  provide 
flexibility  of  revision  when  task  change,  and  (5)  to  be  used  in  conjunction  with  training  to  decrease  training  time. 
Evolving  rapidly,  job  aids  have  taken  on  a multitude  of  shapes,  sizes  and  forms.  Utilizing  a more  formal  approach  to 
the  development  of  job  aids,  Rakow  (1981)  suggests  the  following  job  aid  typology;  example,  cueing,  association, 
procedural ized,  and  analog  classification. 

In  recent  years,  the  advancement  of  cognitive  psychology  has  expanded  the  way  individuals  conceptualize 
job  aids.  Viewed  in  the  past  only  as  external  mechanisms  which  store  information  and  prompt  memory  to  improve 
performance,  job  aids  now  have  taken  on  additional  rolls  such  as  guiding  perspectives,  decisions  and  self  evaluation 
(Rossett,  1991).  Other  individuals  such  as  Tillman  (1985)  and  Sleg  (1988)  suggest  that  job  aids  be  incorporated  as 
an  instructional  intervention  to  assist  in  the  acquisition  of  skills  and  knowledge.  Unfortunately,  this  paradigm  shift 
has  not  occurred  in  the  scientific  community.  Only  a limited  number  of  empirical  research  studies  have  been 
conducted  using  job  aids.  These  studies  have  resulted  in  a contribution  of  valuable  knowledge  in  the  areas  of:  job 
aid  and  improved  performance  (Keefer,  1986;  Lafleur,  1994);  job  aid  development  and  implementation  efforts 
(Ruyle,  1991;  Ford,  1994);  expertise  transfer  using  knowledge  based  systems  as  job  aids  (Stevenson,  1990);  and 
helpfulness  of  job  aids  in  journal  writing  (Cole,  1993).  Little  research  has  been  conducted  to  identify  the  potential 
educational  value  of  job  aids  in  the  domain  of  knowledge  acquisition.  No  research  has  been  conducted  to  identify 
how  prior  knowledge  interacts  with  job  aids  in  the  facilitation  of  knowledge  acquisition.  For  this  reason,  an 
exploratory  investigation  was  undertaken  in  this  unknown  area. 


45 


46 


Statement  of  the  Problem 


Literature  supports  the  use  of  job  aids  to  decrease  efficiency  (finite  resources)  and  improve  effectiveness 
(performance).  But  most  literature  indicates  that  job  aids  are  used  to  recall,  guide  knowledge  and 
procedures  previously  acquired.  Limited  research  has  been  conducted  using  job  aids  as  an  instructional  intervention. 
It  was  hypothesized  that  the  use  of  job  aids  as  an  instructional  intervention  administered  directly  after  interacting 
with  the  instructional  content  would  function  to  facilitate  the  acquisition  and  retention  of  the  content  material.  The 
study  focused  on  the  following  research  questions: 

• Are  all  types  of  job  aids  equally  effective  in  facilitating  learner  achievement  of  different  kinds  of  learning  objectives? 

• Are  different  types  of  job  aids  equally  effective  in  facilitating  learner  achievement  among  learners  identified  as  possessing 
high  or  low  prior  knowledge? 

• Is  there  an  interaction  between  learners'  level  of  prior  knowledge  and  type  of  job  aid? 

• Do  learners  identified  as  high  and  low  prior  knowledge  levels  leam  equally  well  from  identical  types  of  job  aids? 

Procedure 

The  physiology  pretest  was  administered  to  nine  hundred  and  three  participants.  Participation  quartiles 
were  calculated.  Participants  in  the  first  quartile  were  identified  as  the  low  prior  knowledge  group;  whereas 
participants  in  the  fourth  quartile  were  identified  as  the  high  prior  knowledge  group.  Participants  in  the  second  and 
third  quartiles  were  eliminated  from  the  study.  Each  participant  of  the  prior  knowledge  groups  was  then  randomly 
assigned  to  one  of  five  treatment  subgroups.  Total  pretest  participation  quartiles  are  displayed  in  Table  1. 


Table  1.  Total  Pretest  Participation  Quartiles 


Percentage 

Quartile 

Pretest  Score  (Number  Correct) 

100% 

Q4 

32-23 

75  % 

Q3 

22-21 

50% 

Q2 

20-  19 

25  % 

2 1 

18-03 

The  cut  off  score  for  the  low  prior  knowledge  group  was  18.  Participants  in  each  of  the  treatments 
possessing  this  score  or  lower  were  classified  as  possessing  low  prior  knowledge.  Combining  all  five  treatments 
together  the  low  prior  knowledge  group  had  a total  of  247  participants.  The  high  prior  knowledge  group  for  each 
treatment  contained  all  participants  who  scored  23  or  above  on  the  pretest.  Combining  all  five  treatments  together 
the  high  prior  knowledge  group  had  a total  of  241  participants.  The  descriptive  statistics  for  the  total  participation 
group,  total  low  prior  knowledge  group  and  total  high  prior  knowledge  group  are  displayed  in  Table  2. 

Table  2.  Total  Participation  Group,  Total  Low  Prior  Knowledge  Group  and  Total  High  Prior  Knowledge 
Group  Descriptive  Statistics 


Statistical  Computation 

Total  Participation 
Group 

Total  Low  Prior  Knowledge 
Group 

Total  High  Prior  Knowledge 
Group 

N 

903 

247 

241 

Mean 

20.05 

15.83 

25.25 

Standard  Deviation 

4.80 

2.27 

2.06 

Variation 

23.00 

5.17 

4.25 

After  identifying  the  low  and  high  prior  knowledge  groups  of  each  treatment,  participants  with  incomplete 
data  element  were  eliminated.  Every  seventh  participant  was  then  eliminated  until  all  treatments  contained  the  same 
number  of  participants  and  the  prior  knowledge  groups  (low  and  high)  were  distributed  equally.  The  study  was 
conducted  with  a sample  of  N=300.  The  low  prior  knowledge  group  consisted  of  150  participants;  high  prior 
knowledge  group  consisted  of  150  participants.  Each  of  the  five  treatments  was  comprised  of  60  participants  with  30 
low  prior  knowledge  and  30  high  prior  knowledge.  Descriptive  statistics  for  the  sample  based  on  low  and  high  prior 
knowledge  groups  are  shown  in  Table  3. 
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Table  3.  Descriptive  Statistics  for  the  Study  Sample  Based  on  Low  and  High  Prior  Knowledge 


Statistical  Computation 

Low  Prior  Knowledge 

High  Prior  Knowledge 

N 

150 

150 

Mean 

15.62 

25.20 

Standard  Deviation 

2.54 

2.07 

Minimum 

3 

23 

Maximum 

18 

32 

Treatments 

The  instructional  content  used  in  both  studies  was  a self-paced  1, 800-word  instructional  module  describing 
the  human  heart,  its  parts  and  the  internal  processes  which  occur  during  the  systolic  and  diastolic  phases  (Dwyer, 
1972;  Dwyer,  1978).  This  content  was  selected  because  of  the  high  reliability  associated  with  its  dependent 
measures  and  because  it  offers  the  ability  to  investigate  different  intellectual  skill  learning  objectives  (facts,  concepts, 
rules  and  problem  solving)  that  are  effective  and  efficient  performance  functions  of  a typical  work  environment.  The 
job  aid  typology  developed  by  Rakow  (1981)  provided  the  skeletal  format  for  each  job  aid  because  each  job  aid  type 
identified  in  his  typology  offers  advantages  when  used  correctly.  In  order  words,  specific  job  aid  types  must  be 
developed  for  specific  results.  To  improve  learner  performance,  it  is  vital  that  the  instructional  design  practitioner 
identifies  the  type  of  information  required  and  how  the  information  will  be  utilized  (Gagne,  Briggs  & Wager;  1972). 
Keeping  this  in  mind,  each  of  the  treatments  developed  for  the  study  mirror  a specific  intellectual  skills  hierarchy 
level.  The  cueing  job  aid  possessed  facts;  the  association  job  aid  possessed  facts  and  concepts;  the  procedural ized 
job  aid  possessed  facts,  concepts,  and  rules/procedures;  and  the  analog  job  aid  possessed  facts,  concepts, 
rules/procedures  and  problem  solving  information. 

The  same  self-paced  instructional  module  was  distributed  to  all  treatment  groups.  In  addition  to  this 
module,  all  but  the  control  group  received  some  type  of  job  aid  and  in  some  cases  two  job  aids  that  were  developed 
by  the  researchers  after  conducting  an  extensive  literary  search,  collaborating  with  a content  expert  and  determining 
reliability  of  the  job  aids  through  a pilot  study.  Job  aids  are  thought  to  be  self-sustaining,  independent  from  training 
and  for  this  reason,  the  researchers  did  not  incorporate  the  job  aids  into  the  instructional  content  but  placed  them  at 
the  end  of  the  instructional  material  to  facilitate  synthesization,  integration  and  recall  of  the  designated  information. 
The  five  treatments  described  in  Table  4 were  developed  specifically  for  the  research  study. 

Table  4.  Treatment  Descriptions 


Treatment 

Number 

Group 

Name 

Description 

1 

Control 

Participants  received  the  instructional  module  containing  no  job  aids. 

2 

Cueing 
Job  Aid 

Participants  received  the  instructional  module  containing  two  job  aids  designed 
to  direct  attention  to  the  facts  but  did  not  provide  step-by-step  directions.  The 
first  job  aid  contained  a heart  graphic  with  structural  parts  labeled  and  the 
second 

job  aid  contained  a heart  graphic  with  labeled  processes  associated  with  blood 
circulation  in  the  heart.  The  information  in  the  cueing  job  aids  provided 
participants  with  specific  information  they  needed  to  discriminate  between 
presented  stimuli. 

3 

Association 
Job  Aid 

Participants  received  the  instructional  module  containing  one  job  aid  which 
related  unknown  information  to  already  known  information  in  the  typical  format 
of  a reference  document.  The  job  aid  contained  facts  using  a heart  graphic  with 
labeled  parts  numbered  and  concepts  in  the  form  of  a definition.  The  numbers 
identified  the  heart  parts  and  their  respective  definitions.  This  job  aid  enabled 
the 

participant  to  identify  a stimulus  as  a member  of  a group  possessing  some 
common  characteristics,  even  though  the  stimuli  would  differ  from  each  other. 

4 

Procedural  zed 
Job  Aid 

Participants  received  the  instructional  module  containing  one  job  aid  describing 
the  twelve  step-by-step  process  of  how  the  blood  circulates  through  the  heart. 
Each  step  had  an  explanation  and  a heart  graphic.  The  job  aid  used  facts, 
concepts 

and  a procedure  associated  with  the  third  intellectual  skill  level. 

5 

Analog 
Job  Aid 

Participants  received  the  instructional  module  containing  two  job  aids  that 
assisted  participants  to  conceptualize  knowledge  of  organization,  structure  and 
relationships  to  solve  problems  and/or  generate  higher  level  concepts  and  values 
necessary  to  solve  problems.  One  analog  job  aid  presented  knowledge  of  heart 
valve  relationship;  the  other  job  aid  provided  knowledge  of  the  circulation  of 
blood  through  the  heart. 

47 

43 


Criterion  Measures 

Four  individual  criterion  measures,  each  measuring  different  learning  objectives,  were  used  as  the  cognitive 
dependent  variables.  The  criterion  measurers  were  administered  immediately  after  the  participants  completed 
interacting  with  their  respective  self-paced  instructional  treatments.  Table  5 summarizes  the  criterion  measures.  The 
listed  descriptions  are  adapted  from  a description  provided  by  Dwyer  (1978,  pp.  45-47)  and  illustrates  the  types  of 
learning  objectives  that  were  assessed  in  this  study.  KR  reliability  coefficients  are  provided  for  each  measure. 

Table  5.  Description  of  Criterion  Measures 


Criterion  Measures 

Description 

Drawing  Test 

Participants  recalled  information  found  in  the  instructional  module  to  construct  a 

simple  line  drawing  of  the  heart.  This  20-item  test  evaluated  the  participants’  ability  to  construct  and/or 

reproduce  content  related  items  in  their  appropriate  context  (KR-20,  0.90). 

Identification  Test 

Participants  recalled  information  presented  in  the  instructional  module  to  identify 

parts  and  positions  of  an  object.  The  test  contained  20  multiple-choice  questions  (KR-20,  0.84). 

Terminology  Test 

The  objective  of  this  20-item,  multiple-choice  test  was  to  evaluate  the  participants’ 
understanding  of  facts,  terms  and  definitions  printed  in  the  instructional  module  (KR-20,  0.77). 

Comprehension  Test 

Consisting  of  20  multiple-choice  items,  this  test  measured  the  participants’  deep 
understanding  of  concepts  related  to  the  functioning  of  the  heart  (KR-20,  0.77). 

Total  Criterion  Score 

The  score  is  the  sum  of  adding  the  correct  number  of  responses  of  the  four  criterion  measures.  This 
score  provided  total  understanding  of  the  instructional  module  (KR-20,  0.94). 

Physiology  Pretest 

This  36-item,  multiple-choice  test  was  developed  to  measure  the  participants’  prior 
knowledge  level  of  human  anatomy  and  physiology  (KR-20,  0.92). 

Design 

The  research  design  was  a randomized  post-test  only  design  (Campbell  & Stanley,  1966).  Participant 
scores  on  the  criterion  measures  (drawing,  identification,  terminology  and  comprehension  tests)  were  identified  as 
the  dependent  variable.  The  independent  variable  was  job  aid  types  (cueing,  association,  procedural ized  and  analog) 
and  prior  knowledge  (low  and  high).  The  study  design,  a 5 X 2 crossed  design,  may  be  represented  as  RSIU  (5 
levels  of  job  aids  and  2 levels  of  prior  knowledge).  Alpha  was  set  at  the  .05  level.  A MANOVA  was  conducted  to 
determine  the  overall  effects  where  significant  F ratios  were  obtained.  Table  6 provides  a summary  of  pretest  and 
dependent  variable  descriptive  statistics. 


Table  6.  Pretest  and  Dependent  Variable  Post  Test  Descriptive  Statistics  Correlation,  Means  and  Standard 
Deviation  (N= 300) 


Pretest 

Drawing 

Test 

Identification  Test 

Terminology  Test 

Comprehension  Test 

Pretest 

1.00 

0.40 

0.38 

0.48 

0.37 

Drawing  Test 

1.00 

0.80 

0.63 

0.61 

Identification  Test 

1.00 

0.64 

0.67 

Terminology  Test 

1.00 

0.74 

Comprehension 

Test 

1.00 

Mean 

20.41 

8.86 

12.62 

9.91 

9.76 

Stan  Dev 

5.33 

5.31 

4.61 

3.68 

4.18 

Minimum 

3 

0 

2 

2 

1 

Maximum 

32 

20 

20 

20 

20 

Table  7 presents  the  mean  achievement  and  standard  deviation  scores  achieved  by  participants  in  each  job 
aid  by  criterion  measure.  Where  as  Table  8 depicts  mean  achievement  and  standard  deviation  scores  achieved  by 
participants  for  low  and  high  prior  knowledge  levels. 
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Table  7.  Mean  Achievement  and  Standard  Deviation  Scores  for  Treatment 


Treatment 

J^'"lest  Mean 

Drawing 
Test  Mean 

Identification 
Test  Mean 

Terminology 
Test  Mean 

Comprehension  Test 
Mean 

jGxfi itrol 

4.333 

10.083 

8.900 

8.650 

Cueing 

9.916 

13.483 

9.666 

9.616 

Association 

10.433 

13.300 

10.550 

9.866 

Proceduralized 

10.033 

13.516 

10.933 

11.150 

Analog 

9.066 

12.716 

9.516 

9.533 

Table  8.  Mean  Achievement  and  Standard  Deviation  Scores  for  Prior  Knowledge  Level 


Prior  Knowlejlge^ 

Drawing 

Identification 

Terminology 

Comprehension 

— ftflean 

Test  Mean 

Test  Mean 

Test  Mean 

Test  Mean 

High 

10.920 

14.253 

11.546 

11.266 

Low 

6.793 

10.986 

8.280 

8.260 

The  multivariate  analyses  (MANOVA)  indicated  that  for  all  variables  an  insignificant  interaction  existed 
between  job  aid  type  and  prior  knowledge  level.  However,  for  all  variables  assessed  simultaneously,  a main  effect 
existed  among  the  different  job  aid  treatments  and  between  the  prior  knowledge  levels.  The  results  of  the  MANOVA 
are  displayed  in  Table  9. 


Table  9.  MANOVA  Test  Criteria  and  F Approximations  for  Overall  Effect  Using  Wilks'  Lambda  Statistic 


Variable 

Value  | 

F 

! Num  DF  I 

Den  DF 

I Pr  > F 

Job  Aid 

0.767878466 

4.95215 

16 

877.437 

0.0001 

Prior  Knowledge 

0.762896938 

22.2994 

4 

287.000 

0.0001 

Job  Aid*Prior 
Knowledge 

0.931476308 

1.28913 

16 

877.437 

0.1965 

Follow  up  analyses  indicated  that  all  four  job  aid  treatments  were  significantly  more  affective  than  was  the 
control  treatment  on  all  four  of  the  criterion  measures.  However,  all  job  aids  were  found  to  be  equally  effective  in 
facilitating  learner  achievement  on  each  of  the  criterion  measures.  Follow  up  analyses  also  indicated  that  learners  in 
the  high  prior  knowledge  level  achieved  significantly  higher  mean  achievement  scores  on  each  criterion  measure 
than  did  learners  in  the  low  prior  knowledge  level. 

Results 


The  multivariate  analyses  (MANOVA)  yielded  an  insignificant  interaction  between  job  aid  type  and  level  of 
prior  knowledge  (Table  9).  As  shown  in  Table  9 for  all  variables  simultaneously,  a main  effect  existed  for  job  aids 
(F=4.95)  and  prior  knowledge  (F=22.23). 

Simultaneous  confidence  intervals  using  the  .05  alpha  level  were  calculated  for  each  criterion  measure  to 
determine  job  aid  treatment  effects.  On  the  drawing  and  identification  measures  all  four  job  aid  treatments  (cueing, 
association,  proceduralized,  and  analog)  were  found  to  be  significantly  more  effective  than  the  control  in  facilitating 
student  achievement.  Insignificant  differences  in  achievement  were  found  to  exist  among  participant  mean 
achievement  scores  on  both  criterion  measures.  On  the  terminology  test  measure,  only  treatments  three  (association) 
and  four  (proceduralized)  were  more  effective  than  the  control  treatment  in  facilitating  student  achievement  however, 
all  job  aid  treatments  were  again  found  to  be  equally  effective.  On  the  comprehension  test  measure,  only  treatment 
four  (proceduralized)  was  found  to  be  significantly  more  effective  than  the  control  treatment  in  facilitating 
achievement.  Insignificant  differences  in  achievement  were  found  to  exist  among  the  four  job  aid  treatments. 

To  identify  the  prior  knowledge  treatment  effects,  simultaneous  confidence  intervals  were  computed  using 
the  .05  alpha  level.  Results  indicated  that  low  prior  knowledge  learners  achieved  significantly  lower  mean 
achievement  scores  on  all  criterion  tests  (drawing,  identification,  terminology,  and  comprehension)  than  did  high 
prior  knowledge  learners. 


Discussion 

The  findings  of  this  study  indicated  that  job  aid  treatments  (T2,  T3,  T4,  T5)  were  all  significantly  more 
effective  in  facilitating  learner  achievement  on  the  drawing  and  identification  measures  than  was  the  control 
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treatment  (Tl)  which  contained  only  the  instructional  module.  A possible  explanation  for  these  results  may  be  that 
all  job  aid  treatments  contained  a graphical  picture  highlighting  or  identifying  the  structural  and  organizational 
components  of  the  heart.  These  graphics  helped  the  learners  organize  information  to  facilitate  acquisition,  retention 
and  subsequent  retrieval  of  the  information.  Research  supports  the  fact  that  long  term  memory  visualization 
increases  performance  (Kolers  & Ostry,  1974;  Light,  Berger  & Bardales,  1975).  Gagne,  Walker  Yekovich  and 
Yekovich  (1993)  suggest  that  declarative  knowledge  (knowing  that  something  is  the  case)  is  stored  in  knowledge 
structures  in  the  form  of  a schema.  Schemas  are  characterized  as  having  variables,  being  organized  hierarchically 
(embedded  within  another),  and  facilitating  inferences.  Within  schemas,  declarative  knowledge  is  represented  in  the 
form  of  a proposition,  image  or  linear  ordering.  A proposition  is  a basic  information  unit  such  as  one  idea  and 
expresses  the  relationships  among  concepts.  According  to  Wanner  (1968)  individuals  do  not  store  knowledge  in 
words,  phrases  or  sentences  but  proposition  form.  The  control  group  (Tl)  participants  receiving  the  instructional 
module  containing  only  a narrative  may  have  developed  propositions  without  the  assistance  of  job  aids.  However, 
participants  who  received  the  instructional  module  containing  job  aids  may  have  developed  propositions  from  the 
instructional  narrative  but  also  enriched  the  schema  by  adding  additional  declarative  knowledge  in  the  visual  form  of 
a heart  graphic  and  linear  ordering  (step  1,  step  2,  step  3,  etc.). 

On  the  terminology  measure,  only  treatment  three  (association)  and  four  (proceduralized)  were  more 
effective  than  the  control  treatment.  The  explanation  for  this  finding  may  be  that  the  association  job  aid  required 
additional  participant  interactivity  with  the  instructional  modules  containing  job  aids.  Participants  looked  at  the 
graphic,  identified  a number  related  to  a specific  heart  part,  identified  the  heart  part,  located  the  definition  of  the 
heart  part  listed  at  the  bottom  of  the  job  aid  below  the  graphic,  read  the  definition  and  related  the  definition  back  to 
the  heart  part  located  on  the  graphic.  Used  as  an  elaboration  learning  strategy  this  activity  influenced  the 
construction  and  integration  aspects  of  the  encoding  process.  Internal  connections  are  constructed  in  working 
memory  between  new  information  and  knowledge  stored  in  long  term  memory.  The  proceduralized  job  aid  used  an 
organizational  strategy  which  not  only  identified  and  related  major  and  minor  information  but  also  assisted  in 
selection  or  the  paying  attention  to  incoming  information  which  was  then  transferred  into  working  memory  and 
construction  or  the  building  of  connections  between  ideas  contained  in  working  memory.  Both  the  association  and 
proceduralized  job  aids  provided  relationship  information  and  required  additional  interaction  with  the  instructional 
content.  Braune  and  Forshay  (1983)  and  Craik  and  Tulving  (1975)  both  suggest  that  it  is  only  through  many 
interactions  with  the  task  that  individuals  seem  to  develop  an  understanding  of  functional  relationships  contained  in 
concepts.  The  cueing  job  aids  contained  factual  information  and  required  very  little  interaction.  The  complexity  of 
the  analog  job  aids  may  have  produced  confusion  or  affected  the  ability  of  the  participants  to  discriminate  relevant 
information  for  non-relevant  information.  Nomprocessed  information  is  not  beatable  and  retrievable  during  the 
assessment  phase  and  therefore,  negatively  impacts  achievement. 

All  job  aids  provided  additional  declarative  knowledge  in  image  form  and  may  be  the  reason  that  all  job 
aids  did  not  differ  significantly  from  one  another.  However,  the  additional  interactivity  of  the  association  and 
proceduralized  job  aids  was  not  enough  to  result  in  a significant  difference  between  the  job  aid  types. 

High  and  low  prior  knowledge  participants  did  not  profit  equally  from  the  different  job  aid  treatments. 
Participants  classified  as  low  prior  knowledge  had  significantly  lower  mean  identification,  drawing,  terminology  and 
comprehension  scores  than  did  high  prior  knowledge  participants.  This  finding  is  consistent  with  the  premise  that 
successful  integration  of  new  information  occurs  when  it  is  related  to  existing  long  term  information  (Bransford  & 
Franks,  1976;  Clifton  & Slowiaczek,  1981;  Davey  & Kapinus,  1985).  High  prior  knowledge  participants  who 
received  the  same  treatments  consistently  scored  higher  on  all  four  criterion  post  tests.  According  to  Weinstein  and 
Mayer  (1986),  if  an  individual  already  possess  prior  knowledge  of  a subject,  it  is  easier  to  add  information,  connect  a 
new  schema  to  make  inferences  and  access  the  schema  information. 

All  treatments  contained  in  the  study  were  developed  as  self-contained  instructional  books  containing 
directions  and  instructional  narrative  and  in  some  cases  job  aids.  No  additional  assistance  was  rendered  to  the  study 
participants.  Therefore,  the  findings  of  the  study  may  support  Jonassen  and  Grabowski  (1993)  who  found  that  high 
prior  knowledge  learners  require  less  instructional  assistance  than  do  low  prior  knowledge  learners  and  Willoughby 
and  Wood  (1994)  who  found  that  restriction  of  prior  knowledge  impacts  the  learners  ability  to  integrate  new 
information  and  makes  instruction  more  difficult  and  increase  the  probability  that  achievement  will  be  affected  in  a 
negative  manner. 

Finally,  the  significant  lower  scores  for  low  prior  knowledge  participants  may  be  the  result  of  their  inability 
to  make  inferences  (Gagne,  Walker  Yekovich  & Yekovich,  1993).  Experts  possess  more  declarative  knowledge  and 
rich  domain  specific  relevant  information  than  do  novices  and  possess  the  ability  to  make  correct  inferences  during 
the  spread  of  activation  in  long  term  memory  when  they  are  unable  to  retrieve  exact  information  (Weinstein  & 

Mayer,  1986).  Novices,  on  the  other  hand,  are  less  likely  to  make  correct  inferences  because  of  limited  knowledge 
representation  in  the  cognitive  domain. 


50 


51 


Conclusion 


The  research  findings  indicate  that  the  type  of  paper-based  job  aid  used  with  a self  paced  instructional 
module  would  be  based  on  the  desired  education  objective.  Also,  if  learners  possess  a high  degree  of  prior 
knowledge,  then  paper-based  job  aids  may  be  used  with  self-paced  instructional  content  to  positively  effect  the  types 
of  cognitive  development  evaluated  in  this  study.  The  findings  also  indicate  that  to  improve  job  aid  effectiveness 
some  form  of  interactivity  such  as  rehearsal,  elaboration,  organizational  or  comprehension  monitoring  strategies  be 
incorporated  into  the  design  of  the  job  aids. 

The  study  was  an  attempt  to  venture  into  an  uncharted  area  of  using  paper-based  job  aids  as  an  instructional 
intervention  to  improve  learner  achievement.  However,  the  findings  do  not  provide  enough  evidence  to  alter  the 
current  practice  of  using  job  aids  as  non-instructional  interventions.  It  is  vital  that  additional  research  be  conducted 
to  substantiate  the  findings  of  this  study  and  investigate  additional  variables  which  may  impact  on  the  use  of  job  aids 
as  instructional  interventions.  Future  research  may  include  but  is  not  limited  to  the  following  list  of  potential 
empirical  research: 

• Replicate  the  current  study  to  substantiate  findings  and  increase  credibility. 

• Replicate  the  study  substituting  Factor  B,  prior  knowledge,  with  time-on-task  classifications  to  determine  if  the  amount  of 
time  devoted  to  the  instructional  content  and  job  aid  alters  the  current  achievement  outcomes. 

• Replicate  the  study  using  a different  instructional  delivery  medium.  Use  of  a computer-based  instructional  module 
containing  job  aids  may  significantly  impact  the  current  achievement  outcomes. 

• Replicate  the  study  using  job  aids  containing  additional  forms  of  learning  strategies. 

• Replicate  the  study  comparing  the  self-paced  instruction  with  instructor-led  instruction. 

• Conduct  a longitudinal  study  to  assess  the  effect  of  knowledge  retention  when  paper-based  job  aids  are  administered  after 
a self-paced  instructional  module. 
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Abstract 

The  purpose  of  this  study  was  to  investigate  the  effects  of  learning  strategy  and  summarization  within  a 
computer-based  chemistry  and  physics  program.  Students  worked  individually  or  in  cooperative  dyads  to 
complete  science  instruction ,*  half  of  them  completed  summaries  over  the  instructional  content  when 
directed  to  do  so.  The  study  examined  the  effects  of  learning  strategy  and  summarization  on  posttest  and 
enroute  performance , attitude , time-on-task , and  cooperative  interaction  behaviors. 

Introduction 

Cooperative  learning  is  an  instructional  strategy  often  used  by  teachers  for  classroom  application.  While 
some  teachers  implement  cooperative  learning  because  they  believe  it  is  effective  in  terms  of  student  achievement, 
many  teachers  use  cooperative  learning  to  solve  problems  of  classroom  logistics.  Teachers  often  group  students 
around  a computer  to  work  together,  a practice  usually  driven  by  the  scarcity  of  computers  in  the  schools.  While  this 
practice  may  be  driven  by  necessity,  teachers  using  limited  resources  must  find  ways  for  students  to  learn  effectively 
while  using  computers  in  group  situations. 

A number  of  research  studies  have  been  conducted  to  investigate  the  effectiveness  of  cooperative  learning. 
The  effects  of  cooperative  learning  have  often  been  studied  in  comparison  to  individual  or  competitive  learning  in 
situations  ranging  from  kindergarten  classrooms  to  courses  offered  on  college  campuses  (Carrier  & Sales,  1987; 
Cohen,  1990;  Hooper,  1992a;  Sharan,  1980;  Slavin,  1980,  1988).  Learner  performance  on  cooperative  tasks  has 
been  measured  using  text-based,  television-based  and  computer-based  instruction  (Dalton,  Hannafin  & Hooper, 
1989;  Klein  & Pridemore,  1992;  Slavin,  Stevens  & Madden,  1988).  Many  studies  have  yielded  positive  results  for 
achievement  and  attitudes  when  cooperative  learning  was  implemented  in  classroom  settings  (Johnson  & Johnson, 
1989;  Sharan,  1980;  Slavin,  1990).  However,  the  overall  results  of  implementing  cooperative  learning  with  media 
such  as  computer-based  instruction  have  been  mixed  at  best. 

Reviews  of  cooperative  learning  research  are  generally  positive  for  achievement  when  cooperative  learning 
is  used  in  classroom  settings.  Slavin  (1987)  reviewed  35  studies  which  met  the  four  conditions  of  cooperative 
learning  and  which  used  group  rewards  based  on  the  sum  of  group  members’  individual  learning;  he  reported  that  in 
30  of  those  studies,  researchers  found  significantly  greater  achievement  for  cooperative  groups  than  control 
conditions.  In  a later  review,  Slavin  (1990)  noted  that  most  cooperative  learning  studies  measure  achievement,  and 
that  more  than  half  of  those  found  significantly  greater  achievement  for  cooperative  learning  groups  than  for  control 
conditions. 

While  the  literature  includes  fewer  studies  on  cooperative  learning  combined  with  computer-based 
instruction  (CBI),  some  studies  have  reported  significant  achievement  results  when  cooperative  learning  was  used 
with  CBI.  Dalton,  Hannafin  & Hooper  (1989)  found  that  eighth-grade  students  who  cooperated  to  complete  a 
computer-based  lesson  on  health  significantly  outperformed  individuals  completing  the  same  lesson.  Johnson, 
Johnson  and  Stanne  (1985)  reported  that  eighth-grade  students  who  worked  cooperatively  to  complete  computer- 
assisted  instruction  performed  better  than  those  who  worked  individually  or  competitively.  Other  researchers  found 
significant  results  for  achievement  when  fourth,  fifth,  and  sixth  graders  used  cooperative  CBI  to  learn  math  content 
(Hooper,  1992b;  Hooper,  Temiyakam,  & Williams,  1993). 

However,  not  all  researchers  have  found  significant  achievement  effects  for  cooperative  CBI  when  it  was 
compared  to  individual  CBI.  Carrier  & Sales  (1987)  found  no  significant  achievement  differences  between  college 
students  who  worked  cooperatively  and  those  who  worked  alone  on  either  an  immediate  or  delayed  posttest.  Webb 
(1987)  reviewed  19  cooperative  computer-based  learning  studies,  and  found  that  there  were  no  significant 
achievement  differences  between  cooperative  groups  and  individuals  for  nine  of  them  and  that  cooperative  groups 
outperformed  individuals  in  only  five  of  those  reviewed. 

Another  area  of  research  focus  has  been  the  effect  of  cooperative  computer-based  instruction  on  affective 
variables.  Many  studies  have  reported  mixed  results  for  attitudes  and  affective  variables  in  cooperative  computer- 
based  instruction.  Dalton,  Hannafin,  & Hooper  (1989)  found  that  attitudes  differ  between  high  ability  and  low-ability 
students,  and  also  between  male  and  female  students  in  cooperative  groups  versus  the  individual  condition.  Doran 
(1994)  found  that  overall  attitudes  toward  CBI  were  positive,  but  that  attitudes  toward  learning  structure  for  college 
students  in  cooperative,  collaborative  and  individual  conditions  were  mixed.  However,  Jones  (1995)  found  that  high 
school  students  working  cooperatively  to  complete  a CBI  lesson  on  math  content  liked  cooperative  learning 
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significantly  more  than  those  working  as  individuals  liked  working  alone.  Crooks  (1995)  found  that  college  students 
in  cooperative  learning  structures  preferred  to  return  to  CBI  significantly  more  than  those  in  the  individual 
conditions.  Johnson,  Johnson,  & Stanne  (1985)  found  that  attitudes  toward  computer-based  instruction  did  not  vary, 
but  that  attitudes  were  significantly  less  positive  for  the  competitive  condition  than  for  the  cooperative  and  individual 
conditions. 

In  addition  to  affective  measures,  several  studies  have  examined  how  students  interact  in  a cooperative 
learning  setting.  To  ensure  appropriate  interaction,  students  are  usually  trained  to  cooperate  by  behaving  in  specific 
ways  in  relation  to  their  partner(s)  and  within  the  instructional  environment.  In  a discussion  of  the  conditions  under 
which  cooperative  dyads  become  productive,  Cohen  (1994)  notes  that  in  CBI  tasks,  students  often  divide  the  labor 
into  roles  such  as  thinkist  and  typist  rather  than  interacting,  and  that  this  can  be  avoided  by  preparing  and  instructing 
students  regarding  the  appropriate  levels  of  interaction  during  cooperation.  In  some  studies,  more  specific 
instructions  on  how  and  when  to  interact  resulted  in  higher  achievement  than  did  less  specific  directions  or  no 
directions  (King,  1991;  Meloth  & Deering,  1992;  O’Donnell,  Dansereau,  Hall,  & Rocklin  1987). 

Research  into  the  types  of  interaction  which  yield  the  best  achievement  results  has  shown  that  behaviors 
such  as  asking  and  answering  questions,  giving  and  receiving  help,  and  giving  elaborated  explanations  have  tended 
to  improve  achievement  for  students  of  all  ability  levels  (King,  1990;  Repman,  1993;  Webb,  1982a,  1982b,  1983). 
Results  showing  that  students  who  interact  well  tend  to  achieve  more  have  prompted  researchers  to  examine 
techniques  for  increasing  or  directing  cooperative  interaction.  Some  studies  have  used  very  elaborate  strategies  for 
directing  and  influencing  student  interactions  during  cooperative  learning  situations.  Other  researchers  have  required 
that  learners  follow  a specific  script  and  assume  roles,  or  required  learners  to  review  or  summarize  key  concepts 
(Dansereau,  1987;  Larson,  et.  al,  1985).  O’Donnell  and  Dansereau  (1993)  found  that  students  who  did  not  take  notes 
over  the  content,  but  who  reviewed  cooperatively  performed  as  well  as  those  who  took  notes;  additionally,  students 
who  expected  to  review  individually  but  instead  reviewed  cooperatively  with  another  student  performed  better  than 
those  who  reviewed  individually.  Skaggs  and  her  colleagues  (1990)  found  that  directing  interaction  to  require  oral 
summarization,  elaboration,  and  other  types  of  processing  of  technical  content  have  impacted  retention  and  recall  of 
information. 

Researchers  have  studied  summarization  as  one  form  of  interaction  during  cooperative  learning.  Findings 
generally  show  that  achievement  is  enhanced  when  cooperative  subjects  are  cued  to  summarize  content  and  listen 
closely  to  detect  errors  or  omissions  in  summaries  (O’Donnell,  Dansereau,  Hall,  & Rocklin,  1987;  Lambiotte,  et.  al, 
1987).  Spurlin,  Dansereau,  O’Donnell,  & Brooks  (1986)  found  that  cooperative  subjects  who  take  turns  performing 
the  roles  of  summarizer  and  listener  score  better  on  posttests  than  those  who  maintain  static  roles.  These  researchers 
also  reported  that  distributed  summary  .cues  have  a positive  effect  on  achievement,  recall,  and  transfer.  Others  who 
have  studied  learner  interactions  based  on  assigned  roles  during  summarization  found  no  overall  performance 
differences  for  role  assignments  or  summary  type  (O’Donnell,  et.  al,  1987).  In  addition,  Sherman  and  Klein  (1995) 
found  that  eighth  graders  who  completed  a cooperative,  computer-based  science  lesson  with  distributed  cueing 
performed  significantly  better  than  those  in  dyads  who  were  not  cued  to  summarize  and  explain  the  material  to  their 
partners. 

The  purpose  of  the  current  study  was  to  investigate  the  effects  of  learning  structure  and  summarization  on 
achievement,  enroute  performance,  and  attitudes.  High-school  students  worked  either  cooperatively  or  individually 
to  learn  science  content  from  a computer-based  lesson.  Half  of  the  students  in  both  learning  structures  were  required 
to  write  summaries  according  to  instructions  embedded  throughout  the  instructional  program  which  directed  them  to 
summarize  the  main  points  of  the  section  just  covered.  These  directions  were  similar  to  those  used  in  other 
cooperative  learning  studies  (Dansereau,  1987;  O’Donnell  & Dansereau,  1993;  Sherman  & Klein,  1995). 

The  instruction,  practice,  and  feedback  were  provided  by  a computer-based  program  designed  to  provide  an 
interactive  learning  environment,  including  an  on-line  periodic  table  of  elements,  an  animation  program  illustrating 
internal  atomic  structure,  and  animations  of  specific  chemical  compounds. 

The  major  research  questions  addressed  in  this  study  were: 

1.  What  is  the  effect  of  cooperative  versus  individual  learning  structure  on  the  posttest  performance,  enroute 
performance,  attitudes,  and  time -on-task  for  subjects  completing  an  interactive  CBI  science  program? 

2.  What  is  the  effect  of  summarization  versus  no  summarization  on  the  posttest  performance,  enroute  performance, 
attitudes,  and  time-on-task  for  subjects  completing  an  interactive  science  CBI  program? 

3.  Does  requiring  dyads  to  write  summaries  during  CBI  influence  cooperative  interaction  behaviors? 

Method 

Subjects 

The  subjects  for  this  study  were  78  ninth-grade  students  from  a large,  public  high  school  in  the  southwest. 
All  students  were  enrolled  in  a freshman  ChemPhysics  science  class  and  had  access  to  a computer  lab  during  the 
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hour  of  their  science  class.  All  students  had  been  taught  how  to  work  together  in  cooperative  situations  by  their 
teachers. 

Materials 

The  materials  for  this  study  comprised  a computer-based  lesson  on  the  periodic  table  of  elements  and  the 
chemical  and  physical  properties  of  elements.  The  lesson  included  a tutorial,  an  on-line  periodic  table  of  elements, 
and  animated  atomic  models.  The  content  of  the  materials  was  based  on  objectives  and  materials  currently  taught  in 
the  ChemPhysics  course.  The  on-line  tutorial  included  an  introduction,  a review  of  prerequisite  knowledge, 
information  on  the  development,  structure  and  practical  use  of  the  table,  practice  questions,  and  feedback.  There 
were  also  several  sets  of  interactive  exercises,  which  required  that  students  access  the  table  to  complete  the  exercises 
correctly. 

The  introduction  to  the  tutorial  provided  directions  on  how  students  should  use  the  instruction,  general 
navigation,  how  to  interact  with  the  on-line  periodic  table  and  animated  atomic  models,  and  how  to  get  on-line  help 
when  necessary.  Another  section  of  the  introduction  covered  the  impact  of  this  program  on  the  students'  course 
grade.  The  instructional  segment  of  the  program  was  divided  into  three  sections.  Students  completed  two  sections  on 
Days  1-3  of  the  study  and  one  section  on  the  last  day.  The  first  section  introduced  students  to  the  program  and 
navigation,  and  included  screens  explaining  that  the  students'  performance  at  the  computer  and  on  the  posttest  would 
count  toward  their  course  grade.  Those  in  the  summarization  condition  also  saw  two  screens  relating  to  the 
summaries;  the  first  screen  contained  a description  of  the  summaries  to  be  written  and  the  other  contained  an 
explanation  of  the  summary  directions  they  would  see  throughout  the  lessons.  This  introductory  section  consisted  of 
20  screens,  five  of  which  are  interactive  pages  which  taught  students  how  access  and  use  both  the  on-line  periodic 
table  and  on-line  help. 

The  actual  ChemPhysics  course  content  was  presented  in  three  lessons.  Each  lesson  began  with  the 
objectives  and  an  overview  for  the  lesson.  For  the  summarization  condition,  students  saw  two  direction  screens  and 
two  input  screens  in  each  section;  there  were  24  summarization  screens  in  all.  There  were  81  instructional  screens  in 
the  program  and  19  selected-response  practice  questions.  Lesson  One  consisted  of  two  sections  of  information  on  the 
development  and  revision  of  the  periodic  table  of  elements.  Lesson  Two  included  two  sections,  which  covered 
chemical  and  physical  properties  of  elements  and  the  Periodic  Law  that  is  the  basis  for  the  organization  of  the  table. 
Lesson  Three  included  two  sections  on  the  element  key  and  the  practical  use  of  the  key  and  table  by  chemists.  Each 
lesson  included  two  sets  of  practice  questions,  and  provided  the  students  with  the  number  correct  and  points  earned 
at  the  end  of  each  section  in  the  lesson. 

Three  types  of  interactive  exercises  were  embedded  within  the  program.  The  first  type  of  constructed- 
response  item  required  subjects  to  use  the  on-line  periodic  table  to  find  specific  properties  of  given  elements;  there 
were  four  sets  of  element  characteristics  exercises,  which  included  four  items  each.  The  information  was  available 
from  the  Periodic  Table,  and  the  computer  did  not  accept  incorrect  responses;  feedback  was  provided  for  each  item 
within  each  set  of  exercises.  The  second  type  of  interactive  exercise  required  students  to  fill  in  the  three  missing 
components  in  an  element  key.  Six  items  requiring  students  to  construct  element  keys  from  information  available  in 
the  periodic  table  were  included;  the  computer  did  not  accept  incorrect  answers.  The  last  type  of  interactive  exercise 
dealt  with  atomic  composition.  In  addition,  there  were  six  atomic  composition  items,  which  required  the  students  to 
build  atoms  of  the  specified  elements  by  placing  protons,  electrons  and  neutrons  on  a graphic  of  an  atom  shell. 
Students  were  allowed  three  attempts  for  each  item.  A total  of  16  interactive  practice  items  were  distributed 
throughout  the  program. 

Subjects  had  navigational  control  throughout  the  program,  with  a few  exceptions.  Subjects  could  not  jump 
ahead  in  the  program,  but  could  return  to  previously-viewed  screens  within  a section  until  they  encountered  a 
summary  direction  screen  (summary  version).  Review  was  not  permitted  once  students  reached  the  selected-response 
practice  questions  at  the  end  of  each  section. 

The  computer  tutorial  was  supplemented  by  an  on-line,  interactive  periodic  table  of  elements,  which 
provided  the  information  normally  available  in  any  standard  periodic  table.  In  response  to  the  subjects'  input,  the 
table  also  provided  more  specific  information  about  each  element  and  group  of  elements.  From  the  table,  subjects 
had  access  to  the  atomic  animations,  which  displayed  the  atomic  composition  of  each  element  (protons,  neutrons, 
and  electrons). 

The  on-line  periodic  table  of  elements  looked  just  like  a typical  periodic  table,  except  that  the  square  for 
each  atom  displayed  only  the  symbol  for  the  atom.  When  a subject  clicked  on  a particular  atom’s  location,  more 
detailed  information  appeared  in  a text  section  just  to  the  right  of  the  actual  table.  This  information  included  the  full 
name  of  the  element,  the  atomic  number  and  atomic  weight,  as  well  as  typical  properties  of  the  element  itself. 
Different  groups  were  highlighted  when  the  subject  clicked  a specific  button  regarding  that  group  or  type  of  element. 
The  subject  could  also  bring  up  detailed  information  about  the  different  types  of  elements  (such  as  noble  gases,  or 
alkali  metals)  by  clicking  on  specific,  labeled  buttons.  The  table  was  accessible  by  the  subjects  when  they  were 
completing  the  tutorial,  the  practice  exercises,  and  interactive  exercises. 
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The  table  also  featured  a search  engine  that  located  elements  when  the  name  of  an  element,  or  partial  name, 
was  typed  in  to  the  search  engine  dialog  box.  Subjects  could  locate  information  about  an  element  even  if  they  did  not 
know  the  chemical  symbol  for  the  element.  Once  the  element  was  found,  the  square  for  that  element  was  highlighted 
and  the  detailed  information  for  that  element  was  displayed.  If  an  entry  could  not  be  located  within  the  table,  a 
message  came  back  stating  that  a match  could  not  be  found  and  suggesting  that  the  subject  check  the  spelling  of  the 
element  name. 

The  periodic  table  of  elements  was  linked  to  an  atomic  animator.  Based  on  the  selection  made  by  the 
student,  the  animator  displayed  the  atomic  structure  of  any  element,  which  was  selected,  or  active  at  the  time  the 
Show  Atom  button  was  clicked.  The  atoms  were  accurately  represented  by  up  to  18  protons,  neutrons,  or  electrons. 
Atoms  with  an  atomic  number  greater  than  18  were  represented  by  the  same  atom  display  as  18,  but  the  actual 
numbers  of  protons,  neutrons,  and  electrons  were  accurately  stated  within  the  diagram.  Up  to  the  atomic  number  of 
18,  the  elements  were  represented  with  accurate  numbers  of  protons  and  neutrons  in  the  nucleus  of  the  atom,  orbited 
by  the  correct  number  of  electrons  displayed  in  the  correct  levels.  For  example,  the  atomic  animation  for  carbon 
(atomic  number  = 6)  showed  six  yellow  protons  and  six  red  neutrons  in  the  nucleus,  with  two  blue  electrons  in  the 
first  level  and  four  more  blue  ones  in  the  second  level.  The  electrons  circled  the  nucleus  continuously  at  a constant 
distance  when  the  animator  was  running.  The  subject  could  stop  the  animation  at  any  time  by  clicking  the  Hide  Atom 
button.  The  animator  could  only  be  launched  from  the  table,  and  not  directly  from  the  tutorial  screens. 

The  atomic  animator  was  also  accessed  directly  from  several  instructional  screens  on  atomic  composition. 
The  students  could  view  an  animation  by  clicking  on  the  buttons  provided  on  related  instructional  screens,  and  were 
then  returned  to  that  same  screen  from  which  the  Hide  Atom  button  was  clicked. 

From  instructional  screens  containing  information  about  or  references  to  compound  composition,  students 
were  directed  to  access  another  animator,  which  displayed  a specific  set  of  compounds  as  the  combination  of  atoms 
graphically  represented.  Subjects  saw  the  selected  compound,  which  was  accompanied  by  a brief  descriptive  text 
message.  Each  compound  animation  displayed  the  actual  atomic  composition  of  the  compound.  For  example,  water 
displayed  one  oxygen  atom  and  two  hydrogen  atoms,  which  were  all  linked  at  their  outer  electron  levels  and  shared 
the  electrons  necessary  to  fill  the  all  outer  levels  of  all  three.  For  each  animation,  a descriptive  compound- specific 
message  was  displayed,  and  the  message  was  visible  until  the  subjects  dismissed  it  with  a mouse-click. 

A summarization  and  a no  summary  (control)  version  of  the  computer  lesson  were  developed.  Both  versions 
included  all  of  the  instructional  elements  described  above.  The  summarization  version  also  included  24  summary 
screens,  12  that  directed  students  to  list  the  main  points  covered  on  the  information  in  the  preceding  instructional 
segment  and  12  where  the  summaries  were  typed  into  text  fields.  Each  section  included  two  summary  screens. 

The  summarization  version  for  the  cooperative  treatment  directed  students  to  work  together  to  summarize 
the  main  points  of  the  information  and  directed  students  by  name  to  type  in  the  summary.  Students  in  cooperative 
dyads  were  directed  by  name  to  take  turns  typing  in  the  summaries.  The  summarization  version  for  the  individual 
treatment  directed  students  to  summarize  the  main  points  and  type  in  the  summary  each  time.  Subjects  in  the  no 
summary  (control)  conditions  received  exactly  the  same  ChemPhysics  instruction,  practice  exercises,  and  interactive 
exercises,  but  they  did  not  see  the  summarization  screens  directing  them  to  recall  and  type  in  summaries  of  the 
information  presented  in  the  computer-based  tutorial. 

Procedures 

This  study  included  four  different  treatment  groups:  cooperative -summary,  cooperative-no  summary, 
individual-summary  and  individual-no  summary.  Twenty  individuals  were  assigned  to  each  summarization  condition; 
in  the  cooperative  conditions,  20  students  were  assigned  to  summarization  condition  and  18  to  the  no-summary 
condition.  Subjects  in  the  cooperative-summary  group  worked  in  dyads  to  complete  the  computer  lesson  and  were 
provided  with  directions  to  summarize  main  points  during  the  instruction.  Those  subjects  in  the  cooperative-no 
summary  group  worked  in  dyads  to  complete  the  lesson  but  did  not  receive  summary  directions.  In  the  individual- 
summary  group,  subjects  worked  alone  to  complete  the  lessons  and  were  provided  with  summary  directions  while 
those  in  the  individual-no  summary  group  completed  the  lessons  but  did  not  receive  these  directions.  All  subjects 
implemented  their  treatment-specific  computer  lesson  over  four  consecutive  school  days. 

Students  in  three  ChemPhysics  classes  were  randomly  assigned  to  either  a cooperative  or  individual 
treatment  condition.  Subjects  assigned  to  cooperative  treatments  were  randomly  assigned  a partner  in  the  same  class. 
Approximately  half  the  subjects  in  each  class  were  randomly  assigned  to  the  summarization  treatment  and  half  were 
randomly  assigned  to  the  no  summary  treatment. 

Subjects  in  the  cooperative-summary  condition  worked  in  assigned  pairs  to  complete  the  computer  lesson. 
These  dyads  were  told  to  work  together  to  complete  the  exercises,  and  that -each  would  be  assigned  the  score 
achieved  by  the  pair  for  correct  completion  of  the  exercises.  At  specific  points  during  the  lessons,  pairs  were  directed 
to  work  together  to  summarize  the  main  points  of  the  section.  They  were  also  directed  to  take  turns  typing  summaries 
into  the  computer,  and  that  their  grades  on  the  exercises  and  the  posttest,  were  part  of  their  semester  grade  for  the 


56 


course.  All  subjects  were  informed  that  they  would  take  the  posttest  individually.  Finally,  each  subject  completed  an 
attitude  survey  immediately  before  taking  the  posttest. 

Subjects  in  the  cooperative- no  summary  condition  worked  in  assigned  pairs  to  complete  the  lessons.  These 
dyads  were  told  to  work  together  to  complete  the  exercises  and  that  each  would  be  assigned  the  score  achieved  by 
the  pair  for  correct  completion  of  the  exercises;  they  were  told  that  their  grades  on  the  exercises  and  the  posttest 
would  be  part  of  their  semester  grade  for  the  course.  All  subjects  were  informed  that  they  would  take  the  posttest 
individually.  Each  subject  completed  an  attitude  survey  immediately  before  taking  the  posttest. 

Subjects  in  the  individual-summary  condition  worked  alone  at  the  computer  to  complete  the  lessons.  At 
specific  points  in  the  instruction,  they  were  directed  by  name  to  summarize  the  main  points  of  the  section. 
Individuals  were  told  to  complete  all  the  exercises  and  that  they  would  be  assigned  a score  for  correct  completion  of 
the  exercises;  they  were  told  that  their  grades  on  the  exercises  and  the  posttest  would  be  part  of  their  semester  course 
grade.  All  subjects  were  informed  that  they  would  take  the  posttest  individually.  Each  subject  completed  an  attitude 
survey  immediately  before  taking  the  posttest. 

Subjects  in  the  individual-no  summary  condition  worked  alone  at  the  computer  to  complete  the  lessons. 

They  were  told  that  they  would  be  assigned  a score  based  on  correct  completion  of  the  exercises  and  that  their  grades 
on  the  exercises  and  the  posttest  would  be  part  of  their  semester  grade  for  the  course.  All  subjects  were  informed  that 
they  would  take  the  posttest  individually.  Finally,  each  subject  completed  an  attitude  survey  immediately  before 
taking  the  posttest. 

Criterion  Measures 

The  criterion  measures  for  this  study  were  a posttest  and  an  attitude  survey.  Interaction  behavior,  time-on- 
task,  and  enroute  performance  were  also  examined.  Achievement  was  measured  by  a 25-item,  paper  and  pencil 
posttest.  All  subjects  completed  the  posttest  individually;  they  were  informed  at  the  outset  that  their  posttest  grade 
would  be  part  of  their  ChemPhysics  course  grade.  The  posttest  covered  all  of  the  objectives  taught  and  practiced 
within  the  computer  lesson,  as  well  as  all  the  objectives  typically  covered  by  the  ChemPhysics  tests  over  this 
segment  of  material  for  the  course.  The  posttest  consisted  of  17  selected  response  items  and  eight  constructed 
response  items.  The  Kuder-Richardson  reliability  estimate  for  the  posttest  was  .65.  The  experimenter  scored  all 
posttests  using  an  answer  key  that  included  correct  answers  and  possible  points  per  item. 

Attitude  was  measured  by  a pencil  and  paper  survey  administered  just  before  the  posttest.  The  survey 
consisted  of  10  items  which  subjects  responded  to  using  a five-point  Likert-type  scale.  Items  measured  the  three 
attitude  components  of  confidence,  satisfaction,  and  continuing  motivation.  The  Cronbach’s  Alpha  reliability  for  the 
attitude  survey  was  .74. 

Subjects’  behaviors  during  the  study  were  recorded  by  trained  observers  using  checklists  for  consistent 
tracking  and  recording;  the  checklist  and  instructions  were  adapted  from  Klein  & Pridemore  (1994).  Subjects  in 
cooperative  conditions  were  observed  for  30-second  intervals  to  determine  the  type  and  frequency  of  interactive 
behaviors.  Each  observer  recorded  interactions  for  all  cooperating  dyads,  watching  each  dyad  for  30  seconds  before 
moving  to  the  next  dyad.  Observers  noted  how  often  a dyad  discussed  the  material  presented,  how  often  members  of 
a dyad  discussed  navigation,  program  requirements  or  task  parameters,  and  how  often  students  in  a dyad  worked 
individually.  Off-task  behaviors  and  conversation  were  also  tracked.  Observers  also  noted  whether  subjects  shared 
keyboarding  and  summarizing  responsibilities.  Prior  to  recording  observations  during  the  study,  the  observers  met  to 
watch  a videotape  of  a pair  of  ninth  graders  working  together  to  complete  the  instructional  program  and  discuss  the 
categories  of  behavior.  There  was  a discussion  about  the  types  of  interaction  behaviors  and  the  observers  practiced 
recording  interactions  as  they  watched  several  segments  of  the  videotape.  Inter-rater  reliability  was  measured  after 
training;  observers  were  shown  a different  segment  of  video  and  were  directed  to  record  all  interaction  behaviors  on 
the  checklist.  Each  observer’s  record  sheet  was  compared  to  the  record  sheet  completed  by  the  researcher.  Inter- 
rater reliability  exceeded  90  percent. 

Enroute  performance  was  measured  using  scores  on  the  19  selected-response  practice  questions.  Subjects  in 
the  cooperative  condition  worked  together  to  complete  the  items,  and  each  received  the  score  achieved  by  the  pair 
working  together.  Individuals  completed  the  same  questions  and  received  credit  for  the  number  completed  correctly. 

Design  and  Data  Analysis 

This  study  was  a posttest-only,  control  group  design.  It  was  a 2 learning  strategy  (cooperative  versus 
individual)  x 2 summarization  (summary  versus  no  summary)  factorial  study.  Both  learning  strategy  and  level  of 
summarization  were  between-subjects  variables.  Analysis  of  variance  (ANOVA)  was  conducted  on  posttest  and 
enroute  performance  scores.  Attitude  scores  were  analyzed  using  MANOVA.  Univariate  analyses  were  performed  on 
individual  survey  items.  Time  data  were  also  analyzed  using  MANOVA  with  instructional  and  practice  time  as  the 
dependent  variables.  Univariate  analyses  were  conducted  on  the  time  spent  on  different  sections  of  the  program. 
Observation  data  gathered  on  interaction  behaviors  for  dyads  were  analyzed  using  T-tests  of  significance. 


Results 

Posttest  Performance 

The  overall  mean  for  the  posttest  was  27.95  (SD  = 3.59),  or  74  percent.  The  mean  posttest  score  for 
subjects  in  the  cooperative  condition  was  26.95  (SD  = 2.88)  and  was  28.58  (SD  = 3.10)  for  those  in  the  individual 
condition.  Posttest  mean  score  was  27.73  (SD  = 4.15)  for  students  in  the  summary  condition  and  28.38  (SD  = 3.14) 
in  the  no  summary  condition.  Analysis  of  variance  (ANOVA)  conducted  on  posttest  scores  revealed  no  significant 
effects  for  any  condition. 

Enroute  Performance 

Enroute  performance  scores  were  reported  as  the  number  of  practice  questions  completed  correctly  for  all 
conditions.  The  overall  mean  score  for  enroute  performance  was  14.22  (SD  = 2.79),  or  75  percent.  The  mean  enroute 
score  for  subjects  in  the  cooperative  condition  was  13.53  (SD  = 1.98)  and  was  14.55  (SD  = 2.84)  for  those  in  the 
individual  condition.  The  enroute  mean  score  was  13.30  (SD  = 2.87)  for  students  in  the  summary  condition  and 

15.17  (SD  = 2.06)  in  the  no  summary  condition.  Analysis  of  variance  for  enroute  scores  (practice  questions)  showed 
a significant  difference  between  students  who  wrote  summaries  and  those  who  did  not,  F (1,55)  = 7.15,  p = .01. 
Examination  of  the  mean  scores  revealed  that  students  in  the  no-summary  condition  (M  = 15.17)  performed  better  on 
enroute  practice  items  than  those  who  summarized  (M  = 13.30). 

Attitude 

Results  indicated  that  subjects  enjoyed  using  the  computer  program  (M  = 2.01)  and  would  like  to  use  computer 
programs  more  often  (M  = 1.86).  Students  generally  reported  a positive  attitude  toward  the  effectiveness  of  the 
exercises  (M  = 2.22),  the  atomic  animations  (M  = 2.08),  and  the  on-line  periodic  table  (M  = 1.89).  Student  attitudes 
toward  the  learning  condition  showed  that  those  who  worked  cooperatively  liked  working  with  a partner  (M  = 2.08), 
and  those  who  worked  individually  liked  working  alone  (M  = 2.28).  M ANOVA  conducted  on  the  attitude  data 
showed  no  significance  for  any  attitude  items  (alpha  = .05). 

Time-on-Task 

Time-on-task  was  broken  out  as  instruction  time  and  practice  time.  The  overall  mean  for  instructional  time 
was  55.59  (SD  = 16.37)  minutes.  The  mean  instructional  time  for  the  cooperative  condition  was  50.73  (SD  = 13.95) 
minutes  and  was  57.89  (SD  = 17.07)  minutes  for  those  in  the  individual  condition.  Instructional  time  means  were 

55.17  (SD  = 16.48)  minutes  for  those  in  the  summary  condition  and  56.02  (SD  = 16.53)  minutes  in  the  no  summary 
condition.  The  overall  mean  for  practice  time  was  32.69  (SD  = 8.91)  minutes.  The  mean  practice  time  for  the 
cooperative  condition  was  31.50  (SD  = 9.55)  minutes  and  was  33.25  (SD  = 8.65)  minutes  for  those  in  the  individual 
condition.  Practice  time  means  were  29.85  (SD  = 7.49)  minutes  for  those  in  the  summary  condition  and  35.62  (SD  = 
9.41)  minutes  in  the  no  summary  condition.  MANOVA  was  conducted  on  time  data  using  instructional  and  practice 
time  as  the  dependent  variables.  MANOVA  revealed  that  summary  condition  had  a significant  effect  on  time-on- 
task,  F(2,54)  = 6.04,  p < .01.  Univariate  analysis  revealed  that  students  who  summarized  spent  less  time  on  practice 
items  than  those  who  did  not  summarize,  F (1,55)  = 1 1.95,  p < .01  (see  Table  7).  Univariate  analyses  also  revealed  a 
significant  interaction  between  learning  strategy  and  summary  condition  F(l,55)  = 6.76,  p < .01.  Follow-up  Scheffe 
multiple  comparison  tests  indicated  that  the  practice  time  mean  for  those  in  the  individual-no  summary  group  (M  = 
34.22)  was  significantly  different  from  the  means  for  those  in  the  cooperative- summary  (M  = 24.99)  group.  The 
Scheff6  test  also  indicated  that  the  practice  time  mean  for  the  cooperative-no  summary  (M  = 38.72)  group  was 
significantly  different  from  the  cooperative-summary  (M  = 24.99)  group. 

Interaction  Behaviors 

Observational  data  were  gathered  during  the  course  of  the  study  for  cooperating  pairs  only.  The  mean  for 
observed  helping  behaviors  was  8.39  (SD  = 4.54)  for  no-summary  and  was  15.10  (SD  = 3.90)  for  summary  condition 
pairs.  The  mean  for  task-related  behavior  was  2.09  (SD  = 1.57)  for  no-summary  and  was  3.00  (SD  = 1.66)  for 
summary  dyads.  The  mean  for  on-task  individual  behaviors  was  2.82  (SD  = 2.97)  for  pairs  in  the  no  summary 
condition  and  was  2.93  (SD  = 2.50)  for  pairs  in  the  summary  condition.  Finally,  the  mean  for  off-task  behavior  was 
1.45  (SD  = 2.02)  for  no-summary  pairs  and  was  3.53  (SD  = 4.93)  for  summary  pairs.  T-tests  run  on  interaction 
behaviors  revealed  that  subjects  in  the  cooperative-summary  condition  exhibited  significantly  more  helping 
behaviors  than  those  in  the  cooperative  no-summary  condition,  t (30)  = 5.59,  p <.01.  Subjects  in  the  cooperative- 
summary  group  also  exhibited  significantly  more  task-related  behaviors  than  those  in  the  cooperative  no-summary 
condition,  t (30)  = 2.55,  p = .016.  Differences  were  not  significant  for  the  two  remaining  interaction  behaviors. 
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Discussion 

The  purpose  of  this  study  was  to  investigate  the  effects  of  learning  strategy  and  summarization  within  a 
computer-based  chemistry  and  physics  program.  Students  worked  individually  or  in  cooperative  dyads  to  complete 
science  instruction;  half  of  them  completed  summaries  of  the  instructional  content  when  directed  to  do  so.  The  study 
examined  the  effects  of  learning  strategy  and  summarization  on  posttest  and  enroute  performance,  attitude,  time -on- 
task,  and  interaction  behaviors. 

Results  indicated  no  significant  differences  for  posttest  performance.  One  possible  explanation  for  this 
result  lies  in  the  power  of  aligned  and  well-designed  instructional  materials.  Subjects  in  this  study  used  a 
competency-based  instructional  package  developed  to  address  each  of  Gagne’s  nine  events  of  instruction.  The 
computer  program  introduced  material  in  prerequisite  order  and  provided  appropriate  practice  and  feedback  (Gagne, 
1985;  Sullivan  and  Higgins,  1983).  It  is  possible  that  the  design  of  the  materials  overrode  the  effects  of  the  potential 
differences,  which  were  investigated  via  the  experimental  conditions.  The  design  of  the  ChemPhysics  materials 
supplemented  by  the  on-line  Periodic  Table  and  atom  animation  supports  the  theory  of  Anderson,  Reder,  and  Simon, 
(1996)  that  “combining  abstract  instruction  with  specific  concrete  examples  is  better  than  either  one  alone”  (pg.  8). 

In  fact,  Anderson,  Reder,  and  Simon,  (1996)  cite  a number  of  studies  that  support  the  contention  that  the 
combination  of  abstract  and  concrete  concepts  is  a powerful  method  of  instruction. 

Other  researchers  have  suggested  that  the  performance  of  individuals  and  cooperative  groups  is  often  equal 
when  well-designed  instruction  is  used  by  all  students.  Bossert  (1989)  indicated  that  studies  often  compare 
cooperative  learning  conditions  to  whole-class  methods,  or  compare  well-designed  cooperative  learning  structures  to 
poorly-designed  individual  situations.  In  studies  utilizing  well-designed  instruction  across  all  conditions,  posttest 
results  do  not  consistently  show  significant  differences  for  cooperative  learning  (Cavalier  1996;  Klein  & Doran, 
1997;  Slavin  & Karweit,  1984;  Slavin,  Madden,  & Leavey,  1984;  Snyder  & Sullivan,  1995). 

Results  also  indicated  no  significant  enroute  performance  difference  between  those  working  in  cooperative 
dyads  and  those  working  alone.  Enroute  items  consisted  of  selected-response  items,  which  were  aligned  to  the 
instructional  content  sections,  presented.  These  items  measured  the  acquisition  of  factual  knowledge,  not  complex 
skills.  Cooperative  learning  researchers  have  found  that  complex  skills  such  as  higher  order  reasoning  provide  the 
best  opportunity  for  cooperative  structures  to  impact  performance  (Bossert,  1989;  Hooper  & Hannafin,  1988).  It  is 
likely  that  the  level  of  processing  or  performance  required  for  students  to  answer  the  selected  response  items  was  not 
sufficiently  demanding  for  those  in  the  cooperative  structure  to  benefit  from  collaboration. 

Another  factor  in  the  lack  of  differences  for  enroute  performance  may  be  due  to  the  consistency  of  the 
computer-based  instruction.  Bossert  (1989)  notes  that  in  many  cooperative  learning  studies,  teachers  often  give 
specific  instructions,  explanations,  encouragement,  and  feedback  to  students  in  the  cooperative  situation  but  do  not 
give  them  to  individual  students.  Computer-based  instruction  provides  consistency  for  students  in  both  cooperative 
and  individual  conditions. 

Results  for  enroute  performance  indicated  that  practice  scores  for  students  who  did  not  write  summaries 
were  significantly  higher  than  for  those  who  wrote  summaries.  It  is  likely  that  the  difference  in  scores  resulting  from 
summary  conditions  was  caused  by  the  amount  of  time  spent  on  practice.  Results  for  time -on-task  showed  that 
students  who  summarized  the  material  spent  significantly  less  time  on  the  practice  items  than  students  who  did  not 
write  summaries.  Students  who  wrote  summaries  may  have  felt  pressured  to  spend  less  time  on  the  practice 
questions,  and  therefore  may  have  been  more  prone  to  answer  incorrectly.  The  lower  enroute  performance  scores  for 
those  in  the  summary  condition  seem  to  indicate  that  those  students  had  insufficient  time  to  complete  the  practice 
items. 

Results  for  time-on-task  also  indicated  a significant  interaction  between  learning  strategy  and  summary 
condition.  Generally,  it  might  be  expected  that  students  working  in  cooperative  dyads  would  take  more  time  to 
complete  practice  items  than  those  working  individually  because  they  had  to  interact  regarding  the  material  and  agree 
on  an  answer.  However,  this  was  only  the  case  for  those  in  the  cooperative  no-summary  condition.  Cooperative  pairs 
in  the  summary  condition  spent  less  time  on  practice  items  than  the  individuals.  Again,  it  is  possible  that  the  time 
limitation  of  class  periods  accounts  for  this  result.  Students  who  had  spent  time  cooperating  throughout  the 
instruction  and  who  then  wrote  summaries  likely  had  the  least  time  remaining  for  the  practice  items.  Cooperative 
students  who  did  not  write  summaries  had  more  time  remaining  for  practice. 

Results  for  time  on  instruction  showed  no  significant  differences  for  students  in  any  of  the  conditions. 
Knowledge  that  summaries  would  be  required  apparently  did  not  impact  time -on-task  for  students  in  either  learning 
strategy  condition.  Although  it  is  generally  expected  that  students  who  cooperate  require  more  time  to  complete 
instruction  (Slavin,  1990),  there  are  some  researchers  who  indicate  that  computer-based  instruction  can  be  as 
efficient  for  those  who  cooperate  as  for  those  who  work  individually  (Klein  & Doran,  1997).  While  it  was  expected 
that  students  in  cooperative  dyads  would  expend  more  time  than  those  who  worked  individually,  the  results  of  this 
study  suggest  that  under  some  conditions,  students  can  learn  as  efficiently  when  cooperating  if  the  instruction  is 
specifically  written  for  cooperative  interaction  to  take  place. 
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Attitude  results  regarding  the  actual  computer-based  program  were  generally  positive.  The  results  of  the 
study  indicate  that  students  enjoyed  learning  about  ChemPhysics  by  using  the  computer  program  and  would  like  to 
learn  more  by  using  other  computer  programs.  This  could  be  reflective  of  the  instructional  quality  and  interactive 
level  of  the  program  itself,  which  encourages  active  involvement  (Bright,  1983;  Caldwell,  1980;  Hannafin  & Peck, 
1988).  The  attitude  results  also  show  positive  student  reactions  to  the  effectiveness  of  the  interactive  exercises,  the 
atomic  animations  and  the  on-line,  interactive  periodic  table.  However,  these  positive  results  may  be  as  much  a 
response  to  the  actual  interactivity  of  those  program  features  as  to  their  effectiveness  as  learning  tools.  It  is  also 
possible  that  these  attitude  results  could  instead  be  attributed  to  a novelty  effect  that  students  experienced  because 
class  was  held  in  the  computer  lab  where  they  used  computers  instead  of  in  the  typical  lecture  and  lab  format  (Bright, 
1983;  Hannafin  & Peck,  1988). 

Attitude  results  regarding  the  learning  structure  from  the  study  were  mostly  consistent  with  positive  attitude 
results  reported  in  other  cooperative  learning  studies.  Attitude  scores  show  that  students  expressed  positive  feelings 
about  the  particular  learning  strategy  to  which  they  were  assigned.  Positive  attitudes  for  cooperative  learning 
structures  have  been  reported  by  a number  of  researchers  (Crooks,  1995;  Jones,  1995;  Sherman  & Klein,  1995; 
Slavin,  1980). 

Results  from  the  study  also  indicated  that  students  in  the  two  cooperative  conditions  interacted  together  in 
somewhat  different  ways.  Student  pairs  in  the  summary  condition  exhibited  more  helping  behaviors  than  those  in  the 
no-summary  condition.  Dyads  in  the  summary  condition  also  interacted  more  regarding  the  instructional  content  of 
the  program.  It  is  likely  that  these  differences  were  the  result  of  the  additional  interaction  required  of  the  summary 
condition,  as  the  students  who  wrote  summaries  had  more  opportunities  during  which  they  could  work  together. 
While  the  observed  interaction  differences  were  statistically  significant,  further  consideration  of  the  actual  means  for 
observed  behavior  indicate  that  the  differences  were  quite  small  because  the  number  of  observed  behaviors  overall 
was  quite  small.  Since  differences  in  observed  interactions  were  small  and  did  not  impact  posttest  performance,  it  is 
important  not  to  over- interpret  the  results  for  interactions. 

This  study  has  some  implications  for  the  application  and  design  of  computer-based  instruction.  It  is  possible 
that  educators  can  compensate  for  the  realities  of  limited  resources  in  schools  by  implementing  well-designed 
computer-based  instruction  with  cooperative  learning.  This  study  indicates  that  computer-based  instruction  designed 
and  developed  following  a systems  approach  may  enable  students  working  together  to  perform  almost  as  well  as 
students  working  alone.  Since  technology  to  deliver  computer-based  instruction  to  individuals  may  not  be 
consistently  available  in  public  schools,  the  availability  of  resources  is  a legitimate  concern  for  teachers  and 
administrators. 

The  current  study  also  suggests  that  CBI  developers  must  consider  practical  time  constraints  prior  to 
including  strategies  such  as  cooperative  learning  or  summarization  in  their  programs.  The  inclusion  of  these 
strategies  may  have  unintended  consequences  such  as  a decrease  in  practice  time  especially  if  students  are  not 
provided  enough  time  to  complete  other  portions  of  the  lesson.  Additional  research  should  explore  the  variables  used 
in  this  study  for  a longer  duration  of  time.  It  might  also  be  worthwhile  to  investigate  the  performance  effects  of 
summarization  during  CBI  in  the  absence  of  other  forms  of  practice. 

Future  cooperative  learning  research  should  also  continue  to  use  well -designed  instruction  that  teaches 
school-age  children  existing  curricular  objectives  (Anderson,  Reder,  and  Simon,  1997).  While  teacher-directed  and 
whole-class  activities  still  dominate  schoolroom  practices,  there  is  considerable  pressure  to  change  to  other  forms  of 
instruction.  Therefore,  there  is  value  in  the  continued  investigation  of  how  to  employ  well-designed  CBI  to  support 
learning  in  small  group  situations. 
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Abstract 

The  purpose  of  this  study  was  to  investigate  the  effect  of  implementing  cooperative  learning  and  objectives 
with  computer-based  instruction  (CBl).  Cooperative  dyads  and  individuals  worked  through  a CBI  earth 
science  program  that  contained  either  instructional  objectives , advance  organizers , or  no  orienting 
activities.  Results  indicated  that  students  who  received  instructional  objectives  performed  significantly 
better  on  intentional  posttest  items  than  students  who  received  either  advance  organizers  or  no  orienting 
activities.  Results  also  revealed  that  dyads  who  received  objectives  exhibited  significantly  more  on-task 
group  behaviors , more  helping  behaviors , and  fewer  off-task  behaviors  than  dyads  in  the  other  orienting 
activity  conditions.  Furthermore , learning  strategy  influenced  time  on  task ; individuals  spent  significantly 
more  time  on  instruction  and  practice  than  cooperative  dyads.  Implications  for  CBI  developers  are 
explored. 

Introduction 

Computer-based  instruction  (CBI)  is  becoming  a more  widely  accepted  instructional  delivery  medium  in 
schools.  However,  access  to  computer  resources  is  limited  in  most  classrooms.  Teachers  who  do  have  computers 
available  usually  have  one  or  two  in  their  classrooms  or  have  access  to  computer  labs  with  fewer  than  15 
workstations  (Becker,  1991).  Working  with  these  restrictions,  teachers  have  limited  options.  They  may  assign 
computer  time  to  individual  students  on  some  type  of  rotation  basis  or  assign  groups  of  students  to  work  at  the 
computer.  Grouping  students  may  be  a practical  solution  to  equipment  shortage.  However,  it  is  not  clear  how 
groups  perform  using  computer  programs  traditionally  designed  for  individual  students. 

While  teachers  frequently  group  students  to  work  together  on  CBI,  software  developers  have  normally 
presumed  that  their  programs  would  be  used  by  individual  students  (Cosden,  1989).  Recently,  a search  of  14 
educational  software  catalogs  revealed  that  only  40  out  of  5,964  CBI  programs  were  designed  with  the  option  of 
implementing  the  program  with  more  than  one  student  at  a time  (Cavalier,  1996). 

Since  teachers  often  group  students  together  to  use  computers,  it  is  important  to  determine  the  factors  that 
influence  learning  and  motivation  in  these  settings.  Several  researchers  have  examined  the  effect  of  implementing 
cooperative  learning  with  CBI.  Some  have  found  positive  effects  for  achievement  and  attitude  when  small  group 
strategies  were  used  with  CBI  (Dalton,  Hannafin,  & Hooper,  1989;  Hooper,  Temiyakam,  & Williams,  1993; 

Johnson,  Johnson,  & Stanne,  1985;  Mevarech,  Silber,  & Fine,  1991;  Trowbridge  & Dumin,  1984).  Others  have 
reported  that  individual  and  cooperative  methods  were  equally  effective  when  used  with  CBI  (Carrier  & Sales,  1987; 
Cavalier,  1996;  Crooks,  Klein,  Jones,  & Dwyer,  1997;  Doran,  1994;  Klein  & Doran,  1997;  Orr  & Davidson,  1993). 

According  to  Klein  and  Pridemore  (1994),  the  mixed  results  for  studies  that  have  examined  cooperative 
learning  with  media  may  be  due  to  the  orienting  activities  employed  within  mediated  lessons.  An  orienting  activity  is 
a mediator  through  which  new  information  is  presented  (Hannafin  & Hughes,  1986).  Orienting  stimuli  evoke 
inspection  behaviors  in  learners,  which  help  to  influence  what  is  learned  from  textual  material  (Rothkopf,  1970).  An 
example  of  an  orienting  activity  is  when  learners  are  provided  with  the  objectives  of  a lesson.  According  to  Gagne 
(1985),  objectives  help  to  activate  a mental  set  that  focuses  student  attention  and  directs  selective  perception  of 
specific  lesson  content. 

Reviews  of  research  have  generally  supported  the  prescription  of  providing  objectives  to  learners. 

However,  inconsistencies  in  the  results  of  these  studies  have  suggested  that  objectives  as  orienting  activities  may  not 
be  effective  in  every  learning  setting  (Duchastel  & Merrill,  1973;  Hamilton,  1985;  Hannafin  & Hughes,  1986; 

Melton,  1978).  Researchers  have  indicated  that  objectives  increase  the  attainment  of  factual  information,  but  do 
little  to  help  students  process  higher  level  skills  (Clark,  1984,  Hannafin,  1985;  Ho,  Savenye,  & Haas,  1986;  Mayer, 
1984).  Others  have  reported  that  objectives  enhance  learning  of  intentional  or  test-relevant  content,  but  provide  little 
assistance  for  learning  incidental  material  (Duchastel,  1972,  1977;  Duchastel  & Brown,  1974,  Kaplan  & Simmons, 
1974;  Morse  & Tillman,  1972;  Rothkopf  & Kaplan,  1972,  1974). 

In  addition  to  objectives,  another  orienting  activity  is  supplying  learners  with  advance  organizers.  Ausubel 
(1968)  defined  an  advance  organizer  as  "relevant  and  inclusive  introductory  materials  . . . introduced  in  advance  of 
learning  ...  at  a higher  level  of  abstraction,  generality,  and  inclusiveness"  (p.  148).  Advance  organizers  relate 
potentially  meaningful  information  to  be  learned  to  existing  structures  that  exist  within  a learner’s  memory;  they 

63 


64 


serve  as  a vehicle  to  assist  learners  to  incorporate  new  information  into  existing  schema  (Ausubel,  1960,  1968). 
Advance  organizers  remind  students  of  something  they  already  know  and  assist  in  organizing  information  to  be 
learned  (Gagne  & Driscoll,  1988).  According  to  Mayer  (1979),  advance  organizers  provide  the  learner  with  a 
framework  that  allows  for  integrative  relationships  to  be  formed  between  new  and  existing  knowledge;  knowledge 
that  is  acquired  goes  beyond  an  isolated  fact  or  concept  and  is  integrated  into  a larger  schema.  Furthermore,  an 
advance  organizer  promotes  learning  when  new  content  is  not  well  organized;  as  the  structure  of  to  be  learned 
material  decreases,  the  greater  the  advantage  of  using  an  advance  organizer  as  an  orienting  activity  (Mayer,  1977, 
1979). 

Researchers  have  found  that  advance  organizers  increase  both  retention  and  comprehension  of  instructional 
content  (Ausubel,  1968;  Mayer,  1984;  Stone,  1983).  But  advance  organizers  have  not  facilitated  performance  in 
every  learning  setting.  Some  researchers  have  indicated  that  advance  organizers  provide  students  with  more  support 
of  incidental  information  rather  than  recall  of  specific  or  intended  learning  (Ausubel,  1978;  Hannafin,  1987;  Mayer, 
1979).  Others  have  reported  that  subjects  given  advance  organizers  outperformed  those  who  were  not  given  this 
orienting  activity  on  material  that  required  broad  assimilation  and  better  understanding;  both  groups  performed 
similarly  on  factual  test  items  (Krahn  & Blanchaer,  1986).  Finally,  the  benefits  of  advanced  organizers  have  been 
reduced  when  more  powerful  instructional  elements  such  as  practice  were  included  in  CBI  lessons  (Hannafin,  1987; 
Hannafin,  Phillips,  Rieber,  & Garhart,  1987;  Phillips,  Hannafin,  & Tripp,  1988). 

The  purpose  of  the  current  study  was  to  investigate  the  effect  of  implementing  cooperative  versus  individual 
learning  and  orienting  activities  during  a CBI  program.  Cooperative  dyads  and  individuals  worked  through  a CBI 
earth  science  program  that  contained  either  instructional  objectives,  advance  organizers,  or  no  orienting  activities. 
While  a number  of  studies  have  been  conducted  to  examine  the  separate  effects  of  cooperative  learning  and  orienting 
activities  during  CBI,  no  research  has  investigated  the  effects  of  both  cooperative  learning  and  orienting  activities 
during  CBI.  A study  by  Klein  and  Pridemore  (1994)  indicated  that  orienting  activities  influenced  student 
interactions  and  knowledge  acquisition  when  students  implemented  cooperative  learning  with  instructional 
television.  The  current  study  was  conducted  to  determine  if  those  results  could  be  extended  to  computer-based 
instruction. 

Method 

Design  and  Subjects 

A 2 X 3 factorial  design  was  used  for  this  study,  with  learning  strategy  (cooperative  versus  individual 
learning)  and  orienting  activity  (objectives,  advance  organizers,  none)  as  the  independent  variables.  The  dependent 
variables  were  performance  and  attitude.  Time  on  task  and  student  interaction  behaviors  were  also  examined. 

Subjects  were  125  fifth  and  sixth  graders  enrolled  at  an  elementary  school  located  in  a lower  socioeconomic 
neighborhood  in  metropolitan  Phoenix.  The  school  was  organized  in  pods  where  grade  levels  were  combined  to 
allow  for  group  and  individual  learning  in  various  subjects  and  special  projects.  The  fifth  and  sixth  grade  students 
who  were  subjects  in  this  study  were  members  of  the  same  pod.  Each  class  of  fifth  and  sixth  graders  was  comprised 
of  approximately  30  students.  Subjects  attended  a weekly  45-minute  class  session  in  a computer  lab  of  28  networked 
Apple  Macintosh  LC  computers.  Prior  to  this  study,  subjects  had  been  taught  basic  computer  skills  such  as  word 
processing  and  presentation  graphics.  Subjects  were  also  experienced  in  using  cooperative  learning  strategies.  They 
regularly  had  participated  in  small  group  projects  in  science,  math,  and  social  science  in  the  classroom.  However, 
subjects  usually  worked  individually  at  the  computer  during  the  weekly  lab  session. 

Materials 

An  original  computer-based  instruction  (CBI)  program  entitled  Prospecting  for  Arizona's  Treasures  was 
developed  for  this  study.  The  program  was  a HyperCard  tutorial  that  presented  information  about  the  tools, 
equipment,  people,  and  methods  used  in  modem  prospecting.  The  content  was  written  at  a fifth  grade  reading  level. 
Six  versions  of  the  CBI  were  developed  with  all  possible  combinations  of  learning  strategy  and  orienting  activity 
used. 

The  tutorial  contained  the  same  content  for  all  versions.  The  program  consisted  of  a total  of  140  screens. 
There  were  20  introductory  screens,  78  information  screens,  34  practice  screens  with  feedback,  5 practice/feedback 
score  screens,  and  3 exit  screens  at  the  end  of  each  lesson.  All  subjects  progressed  through  all  screens  sequentially 
while  controlling  the  pace  of  the  lesson.  The  tutorial  consisted  of  an  introductory  section  and  three  lessons. 

The  introductory  section  consisted  of  five  parts:  identification  information  solicited  from  subjects, 
motivational  information,  navigation  information,  cooperative  or  individual  instructions,  and  the  appropriate 
orienting  activity.  Subjects  were  prompted  to  enter  an  assigned  identification  number  and  their  first  name.  These 
data  were  used  by  the  program  to  address  each  student  by  name  and  assign  tasks  throughout  the  program. 
Motivational  screens  presented  reasons  for  studying  prospecting  in  Arizona.  The  content  of  these  screens  focused  on 
the  impact  of  the  mining  industry  in  Arizona.  Information  and  practice  on  button  operation,  function,  and  navigation 
was  also  presented  in  the  introduction.  As  subjects  practiced  navigation,  they  were  informed  that  a posttest  would  be 
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administered  at  the  conclusion  of  the  CBI  and  extra  credit  points  would  be  earned  by  those  who  attained  a posttest 
score  of  80%  or  more.  These  screens  also  instructed  subjects  in  the  cooperative  groups  to  discuss  all  information 
and  practice  exercises,  and  assigned  roles  and  responsibilities  to  each  dyad  member.  The  final  screen  in  the 
introduction  displayed  the  appropriate  orienting  activity  for  Lesson  1,  Unit  1. 

Lessons  1 and  2 consisted  of  two  units  each.  Lesson  3 consisted  of  one  unit.  Each  unit  had  an  orienting 
activity,  information,  practice,  and  feedback.  Two  versions  of  the  program  contained  instructional  objectives,  two 
versions  contained  advance  organizers,  and  two  versions  contained  no  orienting  activities. 

Objectives  were  presented  as  verbal  information  outcomes  that  students  were  expected  to  possess  after 
instruction  (Gagne,  1985).  Each  unit  included  3-5  objectives.  Each  objective  was  listed  on  a separate  line 
preceded  by  a bullet  to  draw  attention  to  them.  Advance  organizers  were  structured  using  the  approach  described  by 
Ausubel  (1968,  p.  148)  as  "materials  presented  at  a high  level  of  abstraction,  generality,  and  inclusiveness  that  can 
serve  as  anchoring  ideas  for  the  information  to  be  learned.  Each  unit  included  an  advance  organizer.  Each  advance 
organizer  was  presented  in  paragraph  format  and  included  55  - 65  words. 

Information  was  presented  in  a story  format  for  all  three  lessons.  A narrator  character  acted  as  guide  by 
introducing  students  to  members  of  a modem  prospecting  team.  Subjects  were  told  that  they  would  accompany  the 
team  on  a search  for  gold,  silver,  and  copper  simulating  a prospecting  trip.  Characters  acted  out  the  storyline,  with 
the  narrator  character  interrupting  the  story  with  comments,  instructions,  and  assignment  of  tasks.  The  units  of  each 
lesson  served  as  a scene  change  for  the  storyline  and  the  narrator  presented  the  appropriate  orienting  activity  at  the 
start  of  each  unit.  Lesson  1,  Unit  1 is  described  below  as  an  example  of  the  structure  of  each  lesson. 

Unit  1 introduced  three  members  of  a modem  prospecting  team  and  the  equipment  that  each  takes  on  a 
preliminary  search  for  metallic  mineral  deposits.  The  search  strategy  had  been  predetermined  by  various  methods 
that  are  discussed  by  the  prospectors  as  the  story  unfolded.  At  the  end  of  the  unit,  the  narrator  presented  a series  of 
five,  short-answer  practice  items.  Practice  items  were  directly  aligned  with  the  objectives  for  each  unit  and  required 
subjects  to  provide  information  about  the  tools,  equipment,  people,  and  methods  used  in  prospecting.  The  tutorial 
was  programmed  to  allow  for  a variety  of  spellings  and  variations  in  subject  responses. 

The  CBI  was  field  tested  prior  to  the  study  with  a small  group  of  fifth  and  sixth  graders  of  average  ability. 
Revisions  were  made  as  needed  to  ensure  transparent  program  navigation  and  clarity  for  subjects.  For  example, 
navigation  instructions  were  added  at  the  beginning  of  each  lesson  rather  than  placing  them  in  the  overall 
introduction.  Furthermore,  several  practice  items  were  reworded  to  improve  clarity. 

The  cooperative  version  of  the  CBI  incorporated  the  elements  of  positive  interdependence,  face-to-face 
interaction,  and  individual  accountability  (Johnson  & Johnson,  1989;  Slavin,  1991).  Positive  interdependence  was 
established  by  assigning  tasks  and  responsibilities  to  dyad  members,  by  cueing  students  to  switch  these  roles,  and  by 
providing  extra  credit  points  when  both  members  of  a dyad  achieved  80%  or  better  on  the  posttest.  Eighty  percent 
was  considered  mastery  level  at  the  subjects’  elementary  school.  Face-to-face  interaction  was  promoted  by 
instructing  dyads  to  discuss  key  information  and  their  responses  to  practice  items.  Individual  accountability  was 
established  by  having  each  member  of  dyad  independently  complete  the  posttest. 

The  individual  version  of  Prospecting  for  Arizona’s  Treasures  differed  from  the  cooperative  version  only  on 
those  screens  which  provided  directions  and  cues  for  the  dyads  to  cooperate.  Individuals  were  provided  the  same 
incentive  to  perform  well  on  the  posttest;  those  subjects  who  achieved  80%  or  more  on  the  posttest  received  extra 
credit  points. 

In  addition  to  the  CBI  lesson,  a script  of  oral  instructions  to  subjects  was  designed  by  the  researchers.  Oral 
instructions  prior  to  each  computer  session  were  the  same  for  all  subjects.  The  script  informed  all  subjects  how  to 
respond  if  they  needed  assistance  with  the  program  and  how  to  enter  their  ID  number  and  name.  Instructions 
pertaining  to  the  program  differed  for  subjects  in  the  cooperative  and  individual  groups.  The  script  for  cooperative 
groups  provided  instructions  on  how  to  cooperate.  These  instructions  informed  subjects  that  they  would  be  working 
with  a partner  at  the  computer,  that  the  program  was  designed  so  that  they  could  help  each  other  perform  well  on  the 
practice  items,  and  that  both  had  specific  tasks  that  they  were  to  perform.  Oral  instructions  also  prepared  subjects 
for  switching  tasks  at  the  beginning  of  each  unit.  For  example,  the  oral  instructions  for  Lesson  1,  Unit  1 were  as 
follows:  “You  will  be  working  with  a partner  at  the  computer.  This  program  was  designed  for  you  to  help  each 
other.  Both  of  you  will  have  jobs  to  do  while  you  go  through  the  program.  You  should  talk  about  what  is  happening 
in  the  story  and  work  together  to  get  the  answers  to  the  practice  questions.  Lesson  1 has  two  units.  At  the  beginning 
of  the  second  unit,  you  will  switch  jobs.  The  program  will  tell  you  when  to  do  this.”  These  instructions  were  similar 
for  each  of  the  three  computer  sessions. 

Procedures 

There  were  six  treatment  groups  in  this  study.  The  treatment  groups  were  cooperative  learning-objectives, 
cooperative  learning- advance  organizers,  cooperative  leaming-no  orienting  activity,  individual  learning-objectives, 
individual  learning-advance  organizers,  and  individual  leaming-no  orienting  activity. 
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Prior  to  assigning  subjects  to  treatment  conditions,  reading  ability  scores  from  the  Iowa  Test  of  Basic  Skills 
(ITBS)  were  obtained.  The  mean  score  and  standard  deviations  for  each  of  the  four  intact  classes  were  45.37  (SD  = 
25.33),  47.56  (SD  = 24.85),  49.26  (SD  = 24.43),  and  53.22  (SD  = 24.64).  In  order  to  achieve  a balance  between 
treatments,  the  classes  with  the  highest  and  lowest  means  were  combined  to  make  one  group,  while  the  two 
remaining  classes  were  combined  to  make  another  group.  Both  of  these  groups  were  then  randomly  assigned  to 
either  the  cooperative  or  individual  condition.  The  mean  score  for  the  cooperative  treatment  group  was  49.30  and 
was  the  mean  score  for  the  individual  treatment  group  was  48.41. 

Reading  scores  were  also  used  to  assign  partners  to  subjects  in  the  cooperative  treatment.  Using  an  overall 
median  score  of  45,  subjects  were  identified  as  either  high  ability  or  low  ability.  One  high  and  one  low  ability 
student  was  randomly  assigned  to  each  cooperative  dyad.  Teachers  reviewed  these  pairings  to  verify  that  each  dyad 
included  a high  and  low  ability  learner.  After  subjects  in  the  cooperative  treatment  were  paired,  each  dyad  was 
randomly  assigned  to  one  of  the  orienting  activity  conditions.  Subjects  in  the  individual  treatment  were  also 
randomly  assigned  to  an  orienting  activity. 

The  study  consisted  of  three  45-minute  computer  sessions  conducted  on  three  consecutive  days.  Each 
group  was  given  two  45-minute  class  periods  to  complete  Lessons  1 and  2,  and  30  minutes  of  the  third  class  period 
to  complete  Lesson  3.  The  remainder  of  the  third  class  period  (15  minutes)  was  spent  completing  a student  attitude 
survey. 

When  subjects  arrived  at  the  computer  lab,  they  were  instructed  to  sit  at  their  assigned  computer  but  not  to 
type  at  the  keyboard.  When  all  subjects  were  seated,  instructions  were  read  to  them.  Subjects  were  told  that  they 
would  be  working  at  the  computer  to  complete  a lesson  and  were  instructed  to  raise  their  hand  if  they  had  a question. 
Cooperative  dyads  were  instructed  to  decide  who  would  keyboard  first  and  informed  that  the  computer  would 
prompt  them  to  change  roles  at  the  beginning  of  each  unit.  Subjects  were  then  instructed  to  launch  the  appropriate 
version  of  the  CBI.  Subjects  in  all  treatments  worked  through  each  lesson  at  their  own  pace.  While  cooperative 
subjects  worked  through  each  lesson,  a trained  observer  recorded  instances  of  interaction  behaviors.  All  cooperative 
dyads  were  observed  during  all  three  lessons.  Upon  conclusion  of  Lesson  3,  all  subjects  individually  completed  an 
attitude  survey  in  the  computer  lab.  Upon  returning  to  the  classroom,  teachers  distributed  a posttest  and  all  students 
individually  completed  it. 

Criterion  Measures 

The  two  criterion  measures  in  this  study  were  a posttest  and  an  attitude  survey.  In  addition,  time-on-task 
and  student  interaction  behaviors  were  measured. 

A 30-item,  short-answer,  paper-and-pencil  posttest  was  used  to  measure  acquisition  of  the  information 
presented  throughout  the  CBI  lesson.  The  posttest  was  divided  into  two  sections;  15  items  addressed  intentional 
learning  and  15  items  addressed  incidental  learning.  Intentional  learning  items  were  directly  aligned  to  the  outcomes 
of  each  lesson  and  were  similar  to  the  practice  items  found  in  the  CBI  program.  Two  examples  of  intentional  items 
are  "Name  three  members  of  a modem  prospecting  team"  and  "What  tool  would  you  use  to  break  off  a piece  from  an 
outcropping?"  Incidental  learning  items  tested  information  that  was  provided  throughout  the  program,  but  not 
directly  practiced  or  aligned  with  the  outcomes  of  each  lesson.  Two  examples  of  incidental  items  are  "Name  two 
reasons  that  a laptop  computer  is  a good  tool  to  take  into  the  field"  and  "Why  do  minerals  fall  to  the  bottom  of  a 
stream?"  Each  section  of  the  posttest  was  worth  20  points.  Individual  answers  to  an  item  were  checked  against  a 
scoring  key  and  points  were  assigned  for  each  response.  An  item  was  worth  one  point  unless  it  required  a multiple 
response.  One  person  scored  all  of  the  items  on  this  test.  The  Kuder-Richardson  internal-consistency  reliability  was 
.88  for  the  entire  posttest. 

A 10-item  Likert-type  survey  was  administered  prior  to  the  posttest  to  determine  student  attitudes.  It 
measured  a subjects  preference  for  cooperative  or  individual  learning,  confidence  level  toward  knowing  what  was  to 
be  learned  at  the  beginning  of  each  lesson  through  the  orienting  activity,  perceived  value  of  practice  as  test 
preparation,  degree  of  difficulty  in  understanding  the  content,  enjoyment  of  working  on  a computer,  use  of  a 
storyline  to  convey  content,  and  interest  in  prospecting  as  a topic.  The  Cronbach  Alpha  internal  reliability  of  this 
attitude  survey  was  .67. 

Interaction  behaviors  for  a sample  of  34  cooperative  dyads  (14  received  objectives,  1 1 received  advance 
organizers,  9 received  none)  were  observed  and  recorded  by  two  trained  observers.  The  classification  of  interaction 
behaviors  was  based  on  research  by  Webb  (1982,  1987)  and  Klein  and  Pridemore  (1994).  These  interaction 
classifications  were  divided  into  four  sets  of  behaviors:  helping,  on-task  group,  on-task  individual,  and  off-task 
behaviors.  Helping  behaviors  included  asking  for  help,  giving  help  when  asked,  and  unsolicited  help.  On-task  group 
behaviors  were  taking  turns  and  group  discussion.  On-task  individual  behaviors  were  assuming  control  and  working 
alone.  Off-task  behavior  included  talking  about  something  unrelated  to  the  lesson  and  nonverbal  actions  such  as 
reading  or  working  on  other  assignments.  Inter-rater  reliability  was  conducted  to  ensure  consistency  of  interaction 
observations.  Reliability  estimates  were  .92  for  helping  behaviors,  .95  for  on-task  group  behaviors,  .91  for  on-task 
individual  behaviors,  and  .96  for  off-task  behaviors. 
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Each  observer  was  centrally  located  among  the  computer  stations.  At  two-minute  intervals,  the  observers 
recorded  the  interaction  behaviors  of  a dyad  and  then  rotated  to  the  next  cooperative  dyad.  Observations  continued 
throughout  each  of  the  three  class  periods  while  dyads  worked  at  the  computer.  Each  dyad  was  observed  five  times 
during  the  study.  Interactions  were  documented  on  the  interaction  criteria  scoring  sheet  for  type  and  amount.  As 
each  behavior  was  observed,  a tick  was  recorded  on  the  score  sheet. 

Embedded  in  the  CBI  was  a tracking  routine  that  provided  raw  data  on  time  spent  on  each  screen  as  subjects 
progressed  through  the  tutorial.  Time  spent  on  orienting  activities  screens,  instructional  screens,  and  practice  and 
feedback  screens  was  captured  for  tabulation.  The  time  spent  on  each  lesson  and  the  overall  time  for  the  CBI  was 
also  collected. 

Data  Analysis 

A separate  3X2  multivariate  analysis  of  variance  (MANOVA)  was  conducted  on  data  for  performance, 
attitude,  and  time  on  task.  The  dependent  variables  for  the  MANOVA  on  performance  were  scores  on  intentional 
and  incidental  posttest  items.  The  MANOVA  for  attitude  used  each  survey  item  as  a dependent  measure.  The 
dependent  measures  for  the  MANOVA  on  time  on  task  were  time  spent  on  orienting  activities,  time  spent  on 
instruction,  time  spent  on  practice,  and  total  time  in  the  program.  A one-way  MANOVA  was  conducted  on 
interaction  behaviors  to  determine  the  effect  of  orienting  activities  on  the  interactions  of  subjects  in  the  cooperative 
treatments.  Interaction  behaviors  were  considered  as  a group-based  measure,  since  a combined  score  was  obtained 
for  subjects  in  each  cooperative  dyad.  Each  MANOVA  was  followed  by  univariate  analyses  if  a significant 
multivariate  result  was  found.  Scheffe  multiple  comparison  tests  were  also  conducted  when  a univariate  test 
indicated  a significant  main  effect  for  orienting  activities.  Alpha  was  set  at  .05  for  all  statistical  tests. 

Results 

Performance 

A 3 X 2 MANOVA  conducted  on  posttest  scores  revealed  a significant  main  effect  for  type  of  orienting 
activity,  F(4,218)  = 3.1 1,  p < .05.  MANOVA  did  not  reveal  an  effect  for  learning  strategy  or  an  interaction  between 
orienting  activity  and  learning  strategy.  Follow-up  univariate  analyses  indicated  that  type  of  orienting  activity  had  a 
significant  effect  on  intentional  posttest  scores,  F(2,l  10)  = 6.35,  p < .01,  ES  = .74,  but  not  on  incidental  scores. 
Scheffe  multiple  comparison  tests  revealed  that  subjects  who  received  instructional  objectives  (M  = 12.30) 
performed  significantly  better  on  the  intentional  portion  of  the  posttest  than  those  subjects  who  received  advance 
organizers  (M  = 10.16)  and  those  who  received  no  orienting  activities  (M  = 9.39). 

Time  on  Task 

A 3 X 2 MANOVA  conducted  on  time  data  revealed  a significant  main  effect  for  learning  strategy,  F(4,76) 

= 6.22,  p < .001.  MANOVA  did  not  reveal  a significant  effect  for  orienting  activity  or  an  interaction  between 
learning  strategy  and  orienting  activity.  Follow-up  univariate  analysis  showed  that  individuals  spent  significantly 
more  time  on  instruction,  F(  1 ,79)  = 14.35,  p < .001,  ES  = .80,  and  practice,  F(l,79)  = 20.36,  p < .001, 

ES  = .92,  than  subjects  working  in  cooperative  dyads. 

Attitudes 

Mean  scores  for  the  attitude  survey  suggested  that  most  subjects  liked  using  the  computer  program  (M  = 
1.94)  and  thought  the  story  made  it  fun  to  learn  (M  = 2.24).  Subjects  indicated  that  the  information  was  easy  to 
understand  (M=  2.08)  and  that  the  practice  was  helpful  (M-  1.82).  Subjects  who  worked  cooperatively  reported  that 
they  liked  working  with  a partner  (M=  2.00)  while  those  who  worked  individually  indicated  that  they  liked  working 
alone  (M  = 2.36).  Furthermore,  subjects  in  all  orienting  activity  conditions  felt  that  they  knew  what  they  were 
supposed  to  learn  from  the  program  (M  = 1.99).  A 3 X 2 MANOVA  conducted  on  the  attitude  data  did  not  reveal  a 
significant  main  effect  for  orienting  activity,  learning  strategy,  or  an  interaction  between  these  variables. 

Interaction  Behaviors 

A one-way  MANOVA  conducted  on  interaction  behaviors  revealed  a significant  main  effect  for  type  of 
orienting  activity,  F(2, 13)  = 5.92,  p < .001.  Follow-up  univariate  analyses  indicated  that  orienting  activity  had  a 
significant  effect  on  helping  behaviors,  F(2,31)  = 5.55,  p < .01,  on-task  group  behaviors,  F(2,31)  = 6.09,  p < .01,  on- 
task  individual  behaviors,  F(2,31)  = 4.58,  p < .05,  and  off-task  behaviors,  F(2,31)  = 8.83,  p < .001.  Scheffe  tests 
indicated  that  cooperative  dyads  who  received  objectives  exhibited  significantly  more  helping  behaviors  and  more 
on-task  group  behaviors  than  those  who  did  not  receive  orienting  activities.  Scheffe  tests  also  revealed  that  dyads 
who  did  not  get  orienting  activities  exhibited  significantly  more  on-task  individual  behaviors  and  more  off-task 
behaviors  than  those  who  received  either  objectives  or  advance  organizers. 


67 


Discussion 


The  purpose  of  this  study  was  to  investigate  the  effect  of  implementing  cooperative  learning  and  objectives 
with  a CBI  program.  Cooperative  dyads  and  individuals  worked  through  a CBI  earth  science  program  that  contained 
either  instructional  objectives,  advance  organizers,  or  no  orienting  activities.  The  study  examined  the  effects  of 
learning  strategy  and  orienting  activities  on  intentional  and  incidental  posttest  performance,  time  on  task,  student 
attitudes,  and  cooperative  interaction  behaviors. 

Results  indicated  that  students  who  received  instructional  objectives  throughout  the  program  performed 
significantly  better  on  intentional  posttest  items  than  students  who  received  either  advance  organizers  or  no  orienting 
activities.  Providing  objectives  at  the  beginning  of  each  computer  unit  most  likely  directed  student  attention  and 
selective  perception  of  relevant  content  (Gagne,  1985).  The  objectives  may  have  helped  students  focus  on  the 
learning  task. 

It  is  also  likely  that  objectives  enhanced  performance  on  intentional  scores  because  the  objectives  provided 
a clear  link  between  expectancies  for  learning  and  incentive  for  learning.  In  this  study,  all  versions  of  the  CBI 
program  informed  students  that  (1)  they  would  take  a test  at  the  end  of  the  program,  (2)  it  was  important  to  learn  all 
they  could  about  prospecting,  and  (3)  they  would  receive  extra  credit  for  achieving  80%  or  better  on  the  test.  This 
information  was  immediately  followed  with  the  appropriate  orienting  activity  for  Unit  1.  Students  who  received  the 
objectives  may  have  seen  a more  direct  relationship  between  incentives  and  expectancies  than  other  students. 

The  results  of  this  study  support  the  work  of  other  researchers  who  have  reported  that  objectives  enhance 
learning  of  intentional  or  test-relevant  content  (Duchastel,  1972,  1977;  Duchastel  & Brown,  1974,  Kaplan  & 
Simmons,  1974;  Morse  & Tillman,  1972;  Rothkopf  & Kaplan,  1972,  1974).  However,  the  current  study  does  not 
corroborate  the  findings  of  those  who  have  suggested  that  advance  organizers  provide  students  with  support  of 
incidental  information  (Ausubel,  1978;  Hannafin,  1987;  Mayer,  1979). 

Advance  organizers  did  not  influence  incidental  learning  in  the  present  study.  This  result  was  likely  due  to 
the  CBI  program  used  by  students.  It  was  designed  following  a systems  approach  and  included  most  of  the  elements 
of  effective  instruction  proposed  by  instructional  design  theorists  (Dick  & Carey,  1996;  Gagne,  1985;  Sullivan  & 
Higgins,  1983).  Mayer  (1977,  1979)  suggested  that  an  advance  organizer  promotes  learning  when  new  content  is  not 
well  organized.  Furthermore,  others  have  found  that  the  benefits  of  advance  organizers  are  reduced  when  more 
powerful  instructional  elements  such  as  practice  are  included  in  mediated  instruction  (Bertou,  Clasen,  & Lambert, 
1972;  Hannafin,  1987;  Hannafin,  Phillips,  Rieber,  & Garhart,  1987;  Klein  & Pridemore,  1994;  Phillips,  Hannafin,  & 
Tripp,  1988). 

It  is  also  possible  that  advance  organizers  did  not  influence  learning  in  the  current  study  because  of  the 
nature  of  the  items  used  to  test  for  intentional  and  incidental  learning.  All  posttest  items  measured  student 
acquisition  of  factual  information  presented  in  the  CBI  lesson.  Other  researchers  have  reported  that  advance 
organizers  do  not  always  increase  the  attainment  of  facts  (Krahn  & Blanchaer,  1986).  In  addition,  Mayer  (1979) 
theorized  that  advance  organizers  help  learners  acquire  knowledge  beyond  an  isolated  fact  or  concept.  Furthermore, 
the  lack  of  effect  for  advance  organizers  in  the  present  study  may  be  due  to  the  design  of  the  organizers  themselves. 
While  the  advance  organizers  were  designed  to  include  "relevant  introductory  materials  at  a high  level  of  abstraction, 
generality,  and  inclusiveness"  (Ausubel,  1968,  p.  148),  little  attempt  was  made  to  relate  information  to  be  learned 
about  prospecting  for  minerals  to  students'  existing  mental  structures  and  schema. 

In  addition  to  performance,  results  of  the  current  study  revealed  that  orienting  activities  had  a significant 
influence  on  the  interaction  behaviors  of  dyads.  Dyads  who  received  objectives  exhibited  significantly  more  on-task 
group  behaviors  and  helping  behaviors  than  those  who  did  not  receive  orienting  activities.  In  addition,  dyads  who 
received  either  objectives  or  advance  organizers  exhibited  significantly  fewer  off-task  behaviors  than  students  who 
did  not  receive  orienting  activities.  These  results  support  Klein  and  Pridemore  (1994)  who  indicated  that  providing 
objectives  to  cooperative  groups  may  be  an  effective  method  for  increasing  student  interactions. 

It  is  possible  that  the  objectives  version  of  the  program  provided  dyads  with  a clear  goal  to  accomplish. 

Most  successful  cooperative  learning  methods  include  some  type  of  group  goal  (Johnson  & Johnson,  1989;  Slavin, 
1990).  Dyads  who  received  objectives  probably  had  a better  understanding  than  other  dyads  of  the  goals  and 
expectations  of  the  CBI  lesson.  According  to  Gagne  (1985)  informing  the  learner  of  objectives  activates 
expectancies  which  “represent  the  specific  motivation  of  learners  to  reach  the  goals  of  learning  that  have  been  set  for 
them"  (p.  78). 

While  orienting  activities  influenced  cooperative  behaviors  in  this  study,  the  performance  and  attitudes  of 
students  in  both  the  cooperative  and  individual  treatments  was  about  the  same.  Again,  these  results  likely  occurred 
because  the  CBI  was  designed  following  a systematic  approach.  Students  may  have  found.the  presentation  of  the 
content  engaging  and  the  structure  of  the  instruction  straightforward  enough  to  overcome  any  preference  for  a 
particular  learning  strategy.  Other  researchers  have  suggested  that  cooperative  learning  studies  which  include  well- 
designed  instructional  materials  do  not  consistently  show  differences  in  favor  of  cooperative  strategies  (Bossert, 
1988-89;  Klein  & Doran,  1997;  Snyder,  1993).  It  is  also  possible  that  performance  and  attitude  were  similar  for 
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individuals  and  dyads  because  the  lesson  focused  on  the  acquisition  of  verbal  information  instead  of  skills  or 
problem  solving. 

While  learning  strategy  did  not  influence  performance  or  attitude,  results  revealed  that  individuals  spent 
significantly  more  time  on  the  CBI  than  dyads.  This  is  somewhat  surprising,  in  light  of  Slavin's  (1990)  review  that 
indicated,  "most  studies  that  have  measured  time  on  task  have  found  higher  proportions  of  engaged  time  for 
cooperative  learning  students  than  for  control  students"  (p.  47).  However,  the  increased  time  for  individuals  may  be 
explained  through  direct  observation  of  students.  Observers  noted  that  individuals  appeared  less  focused  than  dyads 
and  were  easily  distracted.  Individuals  seemed  to  exhibit  more  off-task  behavior  and  thus  spent  more  time  on  the 
program.  In  contrast,  cooperative  dyads  seemed  to  be  more  attentive  to  task.  Dyads  appeared  to  have  little  difficulty 
working  cooperatively  at  the  computer  and  demonstrated  an  efficiency  of  task.  Thus,  dyads  spent  less  time  than 
individuals  on  each  screen.  These  findings  suggest  that  cooperative  dyads  who  have  experience  working  in  groups 
are  likely  to  be  more  efficient  with  their  learning  than  students  working  alone  at  the  computer.  However,  the  fact 
that  dyads  were  formally  observed  and  individuals  were  only  informally  observed,  could  account  for  dyads  being 
more  attentive  to  task  than  individuals. 

The  present  study  has  some  implications  for  practitioners  who  develop  and  implement  CBI.  Results 
strengthen  the  prescription  of  providing  students  with  instructional  objectives  during  CBI  to  increase  learning  of 
relevant  information.  Furthermore,  findings  support  the  use  of  objectives  for  increasing  the  interactions  of  students 
who  work  together  during  CBI  lessons.  Providing  objectives  to  students  may  reinforce  the  structure  and  goals  of  a 
CBI  lesson  even  when  it  is  well-organized  and  includes  other  elements  of  effective  instruction.  Results  also  suggest 
that  teachers  faced  with  limited  resources  can  assign  students  to  work  together  at  a computer.  While  cooperative 
learning  may  not  always  increase  student  achievement  or  improve  attitude,  this  study  illustrates  that  cooperative 
learning  may  be  as  effective  as  individual  learning  when  students  use  well-designed  CBI. 

Since  teachers  often  group  students  together  to  use  computers,  it  is  important  for  educational  technologists 
to  determine  the  factors  that  influence  outcomes  in  these  settings.  Future  research  should  continue  to  investigate  the 
design  elements  that  influence  learning  and  motivation  when  students  work  together  to  implement  computer-based 
instruction.  Future  research  should  also  continue  to  explore  the  use  of  well-designed  computer  instruction  in 
classroom  environments.  As  computer  resources  become  more  available  in  schools,  educational  technologists  must 
continue  to  find  ways  to  maximize  the  effect  that  computer-based  instruction  has  on  learning. 
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Abstract 

Behaviorism  was  the  dominant  epistemology  in  education  approximately  20  years  ago,  while 
Constructivism  has  most  recently  been  the  dominant  epistemology  in  the  field.  In  this  paper,  differences 
between  the  assumptions  of  behaviorism  and  constructivism  are  briefly  identified,  and  subsequently ; the 
behaviorist  pedagogy  of  Individually-Guided  Education  and  the  constructivist  pedagogy  of  Problem-Based 
Learning  are  analyzed  across  six  core  elements  of  the  instructional  environment,  including  learning  goals, 
instructional  approaches,  teacher  s role,  students'  roles,  resource  utilization  and  evaluation  strategies.  In 
conclusion,  the  authors  discuss  the  similarities  in  pedagogy  resulting  from  two  very  different 
epistemologies. 


Introduction 

Behaviorism  was  the  dominant  epistemology  in  education  during  the  1970s  and  ‘80s,  while  constructivism 
is  currently  the  dominant  epistemology  in  the  field.  Although  they  are  different  epistemologies,  some  of  resulting 
pedagogical  approaches  have  points  in  common  and  are  worth  further  discussion.  Under  the  underpinning  of 
behaviorism,  individualized  instruction  was  developed  in  the  1960s  and  gradually  reached  its  peak  in  the  end  of 
1970s.  Individually-Guided  Education  (IGE)  is  one  represented  approach  to  individualized  instruction.  On  the  other 
hand,  the  most  common  approach  that  reflects  the  constructivist  epistemology  is  Problem-Based  Learning  (PBL), 
which  was  originally  applied  in  medical  schools  in  1970s  and  was  further  developed  by  constructivists  in  the  1990’s. 

In  this  paper,  the  assumptions  underlying  behaviorism  and  constructivism  will  be  briefly  outlined  in  respect 
to  their  particular  views  related  to  reality,  knowledge,  the  nature  of  learning  and  the  purpose  of  instruction.  Secondly, 
IGE  and  PBL  approaches  will  be  discussed  as  they  relate  to  six  key  elements  of  instruction. 

Assumptions  of  Behaviorism 

The  behaviorist  epistemology  is  grounded  in  objectivism,  which  assumes  that  there  is  a single  reality 
external  to  individuals.  Based  on  this  objectivist  worldview,  behaviorists  argue  that  learners  ’’acquire”  knowledge 
from  outside  resources  by  engaging  in  learning  activities  (Ertmer  & Newby,  1993).  Ertmer  and  Newby  describe  the 
behaviorist  position  as  one  in  which  knowledge  results  when  a learner  "finds”  it  somewhere  outside  himself,  and 
therefore,  learning  must  be  seen  from  the  behaviorist  view  as  a process  of  actively  setting  about  to  acquire 
knowledge.  In  order  to  help  learners  acquire  knowledge,  the  behaviorist  assumes  that  teachers  will  engage  in 
instruction,  which  from  this  view  is  seen  as  the  process  of  providing  knowledge  to  learners. 

Assumptions  of  Constructivism 

Constructivists  argue  that  there  are  multiple  realities  constructed  by  individuals.  The  human  mind  does  not 
copy  reality  from  outside  directly,  rather,  it  constructs  reality  (Driscoll,  1994).  In  other  words,  there  is  no  shared 
reality.  Constructivists  believe  that  learners  actively  construct  their  own  knowledge  as  they  try  to  make  sense  of  their 
experience  (Driscoll,  1994).  Knowledge  is  constructed  when  an  individual  creates  meaning  from  his  own  experience 
through  social  negotiation  and  through  his  evaluation  of  the  viability  of  individual  understandings  (Savery  and 
Duffy,  1996).  Constructivists  view  learning  as  a construction  process  that  is  context  dependent.  Learners  engage  in 
active  knowledge  construction  and  interpretation  in  order  to  develop  understandings  of  relationships  and  phenomena 
in  the  world  (Brooks  and  Brooks,  1993;  Bender,  Cunningham,  Duffy,  and  Perry,  1995,  Duffy  and  Cunningham, 

1996).  In  constructivist  learning  environments,  learners  should  be  provided  with  opportunities  to  experience 
cognitive  conflict.  This  is  also  called  "puzzlement,”  which  will  help  them  to  develop  new  schema  or  understanding 
through  the  interpreting  process.  Duffy  and  Cunningham  (1996)  describe  learning  as  “an  active  process  of 
constructing  rather  than  acquiring  knowledge,  and  instruction  as  a process  of  supporting  that  construction  rather  than 
communicating  knowledge." 

Differences  between  Behaviorism  and  Constructivism 

Based  on  the  assumptions  stated  above,  one  may  draw  several  conclusions  about  the  differences  between 
behaviorist  and  constructivist  epistemologies.  Where  behaviorism  views  knowledge  as  resulting  from  a finding 
process,  constructivism  views  knowledge  as  the  natural  consequence  of  a constructive  process.  Where  behaviorism 
views  learning  as  an  active  process  of  acquiring  knowledge,  constructivism  views  learning  as  an  active  process  of 
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constructing  knowledge.  Finally,  where  behaviorism  views  instruction  as  the  process  of  providing  knowledge, 
constructivism  views  instruction  as  the  process  of  supporting  construction  of  knowledge. 

Both  behaviorism  and  constructivism  have  spawned  particular  pedagogical  approaches.  Individually  Guided 
Education  was  one  of  the  preeminent  approaches  to  grow  out  of  behaviorism,  while  Problem-Based  Learning  has 
been  one  of  the  most  popular  approaches  to  grow  out  of  the  Constructivist  movement.  Each  pedagogical  approach 
has  imbedded  within  it  recommendations  regarding  six  core  elements  of  the  instructional  environment,  including  1) 
learning  goals,  2)  instructional  approaches,  3)  teacher’s  role,  4)  students’  roles,  5)  the  role  of  resources  and 
technology  to  support  learning,  and  6)  use  of  evaluation  strategies.  In  this  section,  IGE  and  PBL  will  be  compared 
across  these  six  core  elements. 

Behaviorist  Pedagogy:  Individually-Guided  Education 

In  Individually  Guided  Education  (IGE),  like  other  approaches  to  individualized  instruction,  teachers 
diagnose  students’  entry  skills,  identify  behavioral  objectives,  and  plan  a unique  and  appropriate  instructional 
program  to  facilitate  a student’s  achievement  of  the  objectives.  Team  teaching,  non-graded  classes,  and  cross-age 
tutoring  are  used  in  the  IGE  model. 

According  to  Saettler  (1990),  the  IGE  system  was  originally  developed  by  Herbert  Klausmeier.  The  U.S. 
Office  of  Education  provided  funds  for  large  scale  implementation  of  IGE  in  1971,  and  by  1976  there  were 
approximately  3,000  elementary  and  middle  schools  using  some  form  of  IGE. 

Learning  Goals 

The  IGE  environment  is  structured  to  support  three  fundamental  goals  regarding  the  individual  learner:  1)  to 
foster  self-determinism,  encouraging  the  individual  to  take  charge  of  his  own  destiny,  2)  to  encourage  self- 
actualization  by  allowing  the  individual  to  act  on  his  own  personal  needs  and  desires,  and  3)  to  support  self-direction 
- one  should  be  free  from  inappropriate  mechanisms  of  control  (Romberg,  1985). 

The  basic  assumption  of  IGE  is  that  the  learning  of  each  individual  child  in  these  regards  must  be  the  focal 
point  of  the  school  (Romberg,  1985).  In  order  to  ensure  that  each  individual  child  learns,  it  is  necessary  to  provide 
curricula  that  are  relevant  to  each  student,  and  to  identify  behavioral  goals  for  each  student  so  that  diagnosis  and 
prescription  can  be  used  to  measure  achievement  of  goals.  In  the  IGE  system,  teacher  teams  ensure  that  appropriate 
goals  for  each  student  are  set,  and  students  are  encouraged  to  achieve  these  goals,  in  order  to  get  feedback  regarding 
progress  toward  goals  (Klausmeier,  1975). 

Instructional  Approaches 

Implementation  of  the  IGE  system  requires  that  instructional  time  be  spent  with  teachers  and  students 
collectively  engaged  in  a cycle  of  exploration  of  individual  students’  learning  styles,  motivation  levels,  and 
appropriate  objectives,  engagement  in  suitable  learning  activities,  and  evaluation  of  the  attainment  of  objectives.  In 
the  IGE  system,  heavy  emphasis  is  placed  on  allowing  variation  in  instructional  approaches,  specifically,  varying  the 
amount  of  attention  and  guidance  provided  by  the  teacher,  varying  the  amount  of  time  spent  in  interaction  among 
students,  and  varying  the  amount  of  time  spent  by  each  student  in  one-to-one  interactions  with  the  teacher, 
independent  study,  small-group  activities,  and  adult-led  large  group  activities  (Klausmeier,  1975). 

Teacher  Role 

In  IGE,  teachers  work  as  cooperative  instructional  teams  in  a multi-unit  school  organization,  in  order  to 
plan,  implement,  and  evaluate  instructional  programs  for  individual  students  in  the  instructional  unit  (which  replaces 
the  traditional  age-graded  classroom  (Jeter,  1981;  Romberg,  1985). 

Teachers’  work  involves  the  organization  of  learning  materials  so  that  students  can  make  effective  progress 
through  their  learning  activities;  arrangement  of  the  learning  space  to  allow  for  students  to  work  alone  or  together  to 
achieve  particular  learning  goals;  keeping  efficient  records  in  order  to  trace  each  student’s  individual  progress;  and 
helping  each  student  through  his  unique  learning  process.  Additionally,  teachers  are  responsible  for  providing 
resources  and  removing  obstacles  so  students  can  learn  efficiently  (Dell,  1972). 

Student  Role 

The  IGE,  environment  is  built  upon  the  assumption  of  individual  differences:  no  two  learners  achieve  at  the 
same  rate;  achieve  using  the  same  study  techniques;  solve  problems  in  exactly  the  same  way;  possess  the  same 
repertoire  of  behaviors;  possess  the  same  pattern  of  interests;  are  motivated  to  achieve  to  the  same  degree;  are 
motivated  to.  achieve  the  same  goals;  are  ready  to  learn  at  the  same  time,  or  have  exactly  the  same  capacity  to  learn 
(Duane,  1973). 

Building  on  these  assumptions,  students  in  IGE  programs  have  some  involvement  in  planning  and  carrying 
out  their  own  organized  program.  In  this  regard,  the  student  is  not  a passive  learner,  but  works  actively  through  many 
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activities  such  as  manipulating  learning  resources,  communicating  with  peers,  and  engaging  in  learning  activities 
(Dell,  1972). 

Use  of  resources 

Resources  play  a critical  role  in  IGE  environments  because  they  facilitate  the  variation  of  instructional 
approaches  that  is  critical  in  order  to  meet  each  individual  students’  learning  needs.  In  the  instructional  programming 
model  for  IGE,  Klausmeier  specifically  states  variation  of  instructional  approach  may  be  addressed  through  the  use 
of  resources  and  technologies  including  printed  materials,  audiovisual  materials,  direct  experiencing  of  phenomena, 
and  the  use  of  space  and  equipment,  (Jeter,  1981). 

IGE  environments  are  structured  so  that  media  are  handled  directly  by  learners  and  are  an  integral  part  of 
the  curriculum.  A wide  variety  of  media  are  provided  because  they  are  the  tools  that  address  individual  differences 
and  various  learning  styles  by  providing  alternative  means  of  achieving  objectives  (Duane,  1973). 

Strategies  for  Evaluation 

IGE  is  just  one  of  the  many  individualized  instruction  approaches  that  brought  popularity  and  legitimacy  to 
the  concept  of  criterion-referenced  evaluation  (as  opposed  to  norm-referenced  evaluation).  The  primary  purpose  of 
criterion-referenced  evaluation  in  IGE  systems  is  to  ascertain  the  individual  learner’s  status  in  relation  to  stated 
learning  goals  and  performance  objectives,  rather  than  to  compare  the  individual  to  the  group.  Types  of  criterion- 
referenced  measures  that  may  be  used  in  IGE  environments  range  from  paper-and-pencil  tests,  performance  tests 
which  require  the  learner  to  demonstrate  capability  on  a particular  task,  work  samples  that  demonstrate  a student’s 
mastery  of  an  objective,  and  formal  observation  by  the  teacher  when  achievement  of  an  objective  is  best  measured  in 
a natural  setting  (Bechtol,  1973). 

Constructivist  Pedagogy:  Problem-Based  Learning 

Traditional  classroom  instruction  has  been  criticized  because,  in  this  situation,  students  are  passive  learners 
and  fail  to  transfer  what  they  learned  in  school  to  daily  life  outside  of  the  learning  environment.  Since  students 
passively  receive  information  from  teachers,  their  motivation  of  learning  and  retention  of  knowledge  are  low.  Based 
on  the  constructivist  view,  the  problem-based  learning  approach  was  begun  in  medical  schools  in  the  mid-1970s  to 
solve  the  problems  of  low  learning  motivation,  knowledge  retention,  and  rate  of  knowledge  transfer.  In  this  section, 
the  PBL  approach  will  be  outlined  as  it  relates  to  six  elements  of  instruction. 

Goal  of  Learning 

The  PBL  approach  is  intended  to  help  students  develop  learning  skills.  In  order  to  learn,  students  must  have 
metacognitive  skills,  self-directed  learning  skills,  and  thinking-  and  problem  solving  skills. 

PBL  educators  are  concerned  about  metacognition  because  they  assume  that  expert  problem  solvers  monitor 
and  control  their  thinking  process  in  order  to  be  most  effective.  Metacognition  may  be  defined  as  thinking  about 
thinking.  It's  the  "executive  function  in  thinking:  pondering,  deliberating,  or  reflecting  on  the  problem  or  situation; 
reviewing  what  is  known  and  remembered  about  the  kind  of  problem  confronted;  creating  hypotheses,  making 
decisions  about  what  observations,  questions  or  probes  need  to  be  made;  questioning  the  meaning  of  new 
information  obtained  from  inquiry;  reviewing  what  has  been  learned,  what  it  all  may  mean  and  what  needs  to  be  done 
next,  etc."  (Barrow,  1988,  p3). 

PBL  is  also  concerned  with  fostering  students'  self-directed  learning  skills  so  that  they  will  automatically 
continue  their  own  learning  for  the  rest  of  their  lives  (Barrow,  1988).  PBL  allows  students  to  determine  what  and 
how  to  learn  on  their  own  in  order  to  ensure  their  lifelong  learning  skills  and  the  ability  to  be  effective  and 
independent  learners. 

In  a PBL  approach,  students  also  learn  to  identify  learning  issues,  develop  strategies,  locate  information 
relevant  to  the  issue,  and  evaluate  their  solution.  This  entire  learning  process  helps  to  develop  problem  solving  and 
thinking  skills  (Savery  and  Duffy,  1996). 

Instructional  approaches 

Constructivists  rarely  use  the  term  “instructional  strategy,”  rather,  they  talk  about  principles  of  learning  that 
can  be  applied  across  content  areas.  The  instructional  principles  which  constructivists  consider  ideal  for  facilitating 
the  student's  knowledge  construction  process  are  apprenticeship,  situated  cognition  and  authentic  learning,  as  well  as 
cooperative  learning  and  social  negotiation. 

In  an  apprenticeship  learning  environment  learners  are  faced  with  a real  world  situation  that  allows  them  to 
learn  from  doing.  The  apprenticeship  environments  "enculturate  students  into  authentic  practices  through  activities 
and  social  interaction"  (Brown  et  al.,  1989).  Because  learning  is  a construction  of  meaning  in  context  and  is  context- 
dependent,  students  learn  best  in  an  authentic  and  cooperative  learning  environment.  The  goal  of  cooperative 
learning  is  to  share  alternative  viewpoints  and  to  challenge  alternative  points  of  view  (Duffy  and  Cunningham, 
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1996).  In  social  negotiation  learning  situations,  students  share  their  experience  with  others  and  expose  themselves  to 
multiple  perspectives  which  help  them  to  "judge  the  quality  of  their  own  solutions  and  to  learn  perhaps  more 
effective  strategies  for  problem  solving"  (Driscoll,  1994). 

Role  of  the  teacher 

The  teacher’s  role  in  a PBL  approach  has  changed  from  the  traditional  information  disseminator  to  an  active 
curriculum  designer  and  learning  facilitator  (Barrows  and  Myers,  1993).  As  a curriculum  designer,  the  teacher 
selects  learning  problems  to  reflect  curriculum  objectives  and  real-world  issues.  As  a learning  facilitator,  he  uses 
instructional  strategies  such  as  reflecting,  coaching,  questioning,  and  modeling  to  perturbate  student’s  thinking  and 
help  them  to  see  the  conflict  in  their  thought,  in  order  to  accomplish  negotiated  meaning  and  solutions. 

The  facilitation  skills  required  of  the  teacher  are  reflecting,  posing  problems,  questioning,  modeling, 
choosing  appropriate  strategies  and  assessing  learning  (Hawley,  Hsu,  McCall,  and  Schuh,  1997).  According  to 
Savery  and  Duffy  (1996),  the  teacher  is  a facilitator  rather  than  a subject  matter  expert.  He  needs  to  develop  student’s 
thinking  and  reasoning  skills  and  help  them  to  become  self-directed  learners.  During  the  metacognitive  learning 
process,  the  facilitator  challenges  students'  level  of  understanding  and  relevance  of  student-generated  issues,  rather 
than  leading  them  to  the  correct  answer.  Kalishman  and  Mennin  (1993)  describe  the  relationship  of  student  and 
facilitator  as  one  in  which  student  and  tutor  work  collaboratively  together,  and  where  the  student  demonstrates  level 
of  mastery  while  the  tutor  provides  supportive  and  corrective  guidance  specific  to  the  task  undertaken. 

Role  of  the  student 

In  a PBL  environment,  students  are  active  learners.  Students  work  in  small  groups  on  a problem  presented 
by  the  teacher.  Students  need  to  internalize  and  reason  through  the  problem  to  identify  the  learning  issues  and  define 
the  objectives.  As  tasks  and  work  schedule  are  assigned  to  each  group  member,  they  start  to  identify  resources  to 
gather  information  related  to  the  learning  issues.  Each  member  brings  back  and  reports  his  findings  for  the  whole 
group  to  analyze  the  data  or  to  shape  their  learning  issues  in  order  to  identify  the  solutions.  Once  students  synthesize 
and  summarize  findings  into  solutions,  they  need  to  conduct  an  assessment  to  see  if  all  the  objectives  have  been 
satisfied  and  to  evaluate  their  performance.  Finally,  they  receive  feedback  from  the  teacher/facilitator  (Gallagher, 
1997;  Savery  and  Duffy,  1996;  West,  1992). 

Use  of  resources 

Resources  play  a crucial  role  in  the  learning  process,  because  students  are  responsible  for  using  data  in 
order  to  solve  problems.  Having  access  to  multiple  resources  helps  students  to  shape  their  learning  issues  and  to 
more  clearly  see  the  problems  they  face  from  different  perspectives.  The  problem  situation  is  pre-designed  by  the 
teaching  team  and  problem  information  is  distributed  to  students  as  needed  through  the  learning  process  (Stepien, 
Gallagher,  and  Workman,  1993).  In  some  cases  students  are  provided  with  a list  of  resources  from  the  teacher,  while 
in  other  situations  students  need  to  identify  possible  resources  by  their  own  (West,  1992).  PBL  requires  students  to 
use  real-world  resources  including  books  and  databases,  local  resources  and  experts,  journals,  and  internet 
consultations  (Boyce,  VanTassel-Baska,  Burruss,  Sher,  & Johnson,  1997).  The  local  resource  or  experts  may  be 
mentors  in  the  surrounding  communities  who  agree  to  talk  with  students.  Regarding  the  use  of  computers,  De  Corte 
(1994)  describes  the  role  of  the  computer  as  a supportive  system  that  allows  more  student-control  and  induces 
effective  forms  of  interaction  and  cooperation. 

Strategies  for  evaluation 

According  to  Barrow  and  Tamblyn  (1980),  evaluation  should  be  an  integral  part  of  the  learning  process  so 
that  students  will  be  aware  of  the  learning  needs  and  continuously  receive  feedback.  Their  idea  corresponds  to  Duffy 
and  Cunningham’s  (1996)  claim  that  assessment  should  be  ’’ongoing,  rather  than  just  being  an  end  of  the  semester 
rating." 

In  respect  to  what  should  be  assessed,  Norman  (1991)  claims  that  the  central  role  of  PBL  is  the  acquisition 
of  problem  solving  skills.  Therefore,  it  is  important  to  evaluate  students’  problem  solving  skills  independent  of 
knowledge.  Duffy  and  Cunningham  (1996)  suggest  the  use  of  peer  and  self-assessment  to  assess  students’  skills  as 
problem  solver,  self-directed  learner,  and  as  team  member.  Norman  suggests  that  essay  exams  and  problem-based 
oral  exams  are  good  tools  to  assess  students’  problem-solving  and  self-directed  learning  skills.  Additionally,  Norman 
suggests  that  students  should  be  evaluated  in  the  context  of  problem  solving,  so  that  contextual  clues  to  the  situation 
are  available. 

Comparison  of  IGE  and  PBL  approaches 

Table  1 identifies  some  similarities,  as  well  as  some  differences,  between  the  IGE  and  PBL  pedagogical 
approaches,  as  they  relate  to  six  key  elements  of  the  instructional  environment.  It  is  important  to  note  that  it  is  not 
possible  to  make  exact  comparisons  between  IGE  and  PBL  for  each  of  these  elements.  Despite  this,  it  would  seem 
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appropriate  to  conclude  that,  though  there  is  a time  span  of  20  years  between  these  approaches,  they  do  encourage 
similar  emphases  related  to  learning  and  instruction.  The  conclusion  section  of  the  paper  specifically  identifies  the 
commonalties  in  these  two  pedagogical  approaches. 

Table  1:  Summary  comparison  of  instructional  elements  as  prescribed  by  IGE  and  PBL  approaches 


IGE 

PBL 

Learning  Goals 

Develop  students’  capabilities  in: 

• Self-determinism 

• Self-actualization 

• Self-direction 

Develop  students’  capabilities  in: 

• Metacognition  skills 

• Self-directed  learning  skills 

• Thinking  skills  and  problem-solving  skills 

Instructional 

Approaches 

Instruction  as: 

• Cooperative  Instructional  Teams 

• Multi-unit  school  organization 

• Variations  in  pace,  structure 

• Cycle  of  objectives  and 
evaluation 

Instructional  principles  regarding: 

• Cooperative  learning  & social  negotiation 

• Apprenticeship 

• Situated  cognition 

• Authentic  learning 

Role  of  the 
teacher 

Teachers  facilitate  learning  through: 

• Meaningful  learning  activities 

• Removing  obstacles  to  learning 
Teaching  activities  include: 

• Assessment  of  learning 

• Exploration 

• Providing  resources 

• Helping  students  through  unique 
learning  situations 

As  a facilitator  to: 

• Show  student  how  to  construct  knowledge 

• Create  an  instructional  environment 
where  learner’s  processes  of  situational 
sense  making  are  enhanced. 

Facilitation  skills  include: 

• Assessing  learning 

• Reflecting 

• Posing  problem 

• Questioning 

• Modeling 

• Choosing  appropriate  strategies 

Role  of  students 

■ Plan  learning  program 

8 Organize  learning  activities 

8 Engage  in  learning  activities 

8 Manipulate  resources 

8 Communicate  with  peers 

8 Define  problem  and  identify  issues 

8 Frame  learning  objectives 

8 Determine  work  schedule 

8 Gather  and  analyze  data 

8 Present  and  discuss  learning  findings 

8 Synthesize  findings  into  solutions 

8 Justify  solution  and  evaluate  performance 

■ 

Use  of 
Resources 

Types  of  resources  include: 

• Print  materials 

• Audiovisual  materials 

• Technological  equipment 

• Direct  experience 

Types  of  resources  include: 

• Books  and  databases 

• Journals 

• Computers  and  Internet 

• Local  resources  and  experts 

Evaluation 

Strategies 

Evaluation  should  include: 

• Criterion-referenced  measures 

• Observation  in  natural  settings 

• Work  samples 

Evaluation  should  include: 

• Problem  solving  skills 

• Self-directed  learning  skills 

• Skills  as  a group  member 

Conclusion 

It  may  seem  counter-intuitive  to  report  that  the  very  different  epistemologies  of  behaviorism  and 
constructivism  lead  to  some  important  similarities  between  IGE  and  PBL  pedagogical  approaches.  Yet,  the 
conclusion  of  this  study  is  that,  regardless  of  differences  related  to  historical  context  and  to  terminology,  there  appear 
to  be  common  pedagogical  prescriptions  between  IGE  and  PBL  as  they  relate  to  six  core  elements  of  instruction. 

With  regard  to  learning  goals,  both  IGE  and  PBL  prescribe  that  curricula  should  in  some  way  be  relevant  to 
individual  learners,  that  learners  should  have  some  reason  to  buy-in  to  the  learning  experience,  and  finally,  that 
building  self-directed  learning  skills  is  a crucial  outcome  of  learning. 

Regarding  instructional  approach,  IGE  and  PBL  both  acknowledge  the  importance  of  students’  active 
involvement  in  the  activities  of  the  classroom. 

With  regard  to  teacher  role,  both  IGE  and  PBL  emphasize  the  teacher’s  role  as  guide  for  students,  and  not 
as  a source  of  knowledge.  The  teacher’s  responsibility  is  to  create  an  environment  where  students  can  do  their  own 
learning. 

With  regard  to  students’  roles,  both  IGE  and  PBL  emphasize  respect  of  individual  differences,  and  therefore 
students’  right  to  have  a say  in  their  own  educational  experience. 

Regarding  resource  and  technology  utilization,  IGE  and  PBL  both  value  the  creation  of  resource-rich 
environments  in  order  to  address  individual  differences  and  to  foster  richer  and  more  complex  learning. 
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Finally,  with  regard  to  evaluation  strategies,  both  IGE  and  PBL  approaches  appear  to  be  united  in  the  view 
that  norm-referenced  evaluation  is  at  best,  an  inappropriate  evaluation  strategy. 

Perhaps  the  findings  of  this  study  indicate  that  good  pedagogical  principles  stand  above  or  apart  from 
epistemological  differences.  It  seems,  then,  that  the  challenge  for  all  educators,  regardless  of  avowed  epistemological 
views,  should  be  to  work  toward  the  creation  of  educational  environments  that  are  based  upon  sound  pedagogy,  as 
modeled  by  both  IGE  and  PBL  approaches. 
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Abstract 

An  abundance  of  books , articles , and  Web  sites  are  available  to  support  novice  designers  in  the  creation  of 
effective  Web  pages.  Unfortunately , a review  of  these  materials  reflects  an  absence  of  research  to  support 
many  of  these  design  guidelines.  While  some  of  the  suggestions  offered  are  based  on  empirical  research , all 
too  often  this  research  was  conducted  in  simulated,  print-based  environments  (Grabinger  dc  Amedeo, 
1985).  In  addition , conclusions  are  frequently  based  on  investigations  of  older  technologies , such  as 
monochrome  and  low  resolution  monitors,  that  are  non- representative  of  current  learning  environments 
(Hannafin  & Hooper,  1989). 

Increasing  emphasis  on  Web-based  delivery  systems  makes  it  imperative  that  research  be  conducted  to 
establish  (or  confirm)  guidelines  that  are  appropriate  for  current  media  and  that  support  end  users.  While  the  content 
of  Web  delivery  is  critical  to  the  learning  outcomes,  visual  appeal  can  be  equally  important  as  a factor  in  learner 
motivation  (Keller  & Burkman,  1993).  This  study  is  a first  attempt  to  describe  user  preferences  in  font  size  and  line 
spacing  of  Web-based  menus.  This  study  investigated  the  following  two  research  questions. 

■ Given  a non  scrolled  menu  list,  does  font  size  or  line  spacing  affect  user  preference  in  a CRT  displayed  Web  menu? 

■ Given  a scrolled  menu  list,  does  font  size  or  line  spacing  affect  user  preference  in  a CRT  displayed  Web  menu? 


Literature  Review 

Many  principles  of  Web  page  design  have  been  transferred  directly  from  previous  research  in  print  or 
television.  Grabinger  (1989)  notes  that  recommendations  concerning  typographic  screen  design  elements  such  as  line 
spacing,  line  length,  leading,  size  of  letters,  font  characteristics,  and  case  of  letters  are  frequently  based  on  folklore 
and  practice  in  the  visual  arts.  Citing  Lynch,  Rimer  (1996)  points  out  that  current  graphic  design  schemes  for  CRT 
displays  often  “emulate  many  aspects  of  the  style  and  organizational  conventions  of  paper  documents”  (p.  6).  In  a 
summary  of  an  analysis  of  the  literature  on  the  use  of  color  in  screen  design,  Misanchuk  and  Schwier  (1995)  caution 
instructional  designers  against  generalizing  from  research  that  was  (a)  conducted  on  older  and/or  obsolete 
equipment,  (b)  based  on  results  from  different  display  media  and  transferred  to  video  display  terminals,  (c)  not 
sufficiently  similar  to  tasks  performed  during  teaching  or  learning,  and  (d)  did  not  specify  intended  use  of  the  screen 
display.  This  study  was  predicated  on  the  assumption  that  the  same  advice  applies  equally  well  to  other  screen  design 
considerations  such  as  type,  font  size,  and  line  spacing  for  Web  pages. 

Winn  (1993)  offers  the  instructional  message  designer  extensive  guidelines  which  are  based  on  the 
typography  research  of  Miles  Tinker  (carried  out  between  1922  and  1967).  Principles  of  type  size  recommended  by 
Winn  include  using  10  point  type  in  19  pica  lines  set  with  two  to  four  point  leading  for  optimal  legibility.  These 
suggestions  have  been  taken  directly  from  print-based  media  for  application  in  computer  screen  design.  While  these 
may  turn  out  to  be  appropriate  guidelines  (Hartley,  1987),  there  is  currently  little  empirical  research  to  support  the 
recommendations. 

Chen  and  colleagues  (1996)  conducted  a review  of  the  literature  and  determined  that  most  of  the  research 
on  type  size  pertains  to  print  and  television  screen  presentation,  with  very  little  research  on  computer  screens  in 
general  or  font  size  in  particular.  Their  research  examined  the  effects  of  font  size  on  readability  (measured  by 
comprehension)  and  user  preference  in  a hypertext  computer-based  instruction  (CBI)  environment.  Results  of  the 
study  indicated  that  readers  lacked  satisfaction  with  font  sizes  of  10  and  12  points  and  clearly  preferred  text  of  size 
14  and  16  points. 

In  reviewing  the  literature  for  this  study,  few  studies  were  found  concerning  differentiation  of  textual 
purpose  in  Web  screen  design;  however,  Steinberg  (1992)  discusses  the  various  functions  of  menus  in  computer- 
based  instruction,  ultimately  noting  that  “the  purpose  of  a menu  is  to  provide  information  and  to  make  retrieval  as 
simple  and  efficient  as  possible  for  the  intended  users”  (p.  7),  by  taking  into  account  user  characteristics  and  the 
nature  of  the  task(s).  Some  researchers  (Coll,  Coll  & Nandavar,  1993;  Grabinger,  1989)  have  begun  to  look  to  the 
Gestalt  psychologists  for  guidance  in  understanding  how  a learner’s  perceptions  of  screen  organization  and  design 
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affect  his/her  preferences  and  ability  to  complete  learning  tasks.  Grabinger  (1989)  points  out  that  it  is  the 
examination  of  combinations  of  textual  elements  that  will  help  us  develop  design  guidelines  that  will  positively  affect 
learner  cognitive  processes.  An  example  of  such  research  is  the  1993  study  by  Coll  et  al.  which  found  that  the 
physical  layout  of  screen  menus,  while  still  an  important  design  consideration,  was  less  crucial  to  the  learning  task 
than  the  conceptual  layout  (e.g.,  categorical  or  alphabetical). 

Welsh,  Murphy,  Duffy  and  Goodrum  (1994),  in  an  investigation  of  link  display  strategy  and  link  density 
within  a hypermedia  learning  environment,  found  that  users  reported  increased  favorable  ratings  on  ease  of  use  when 
submenus  were  incorporated  in  the  page  design  (in  white  area[s]  outside  the  body  of  text)  compared  to  iconic  and 
textual  link  indicators  placed  within  the  text.  ‘This  finding  may  be  attributable  to  the  added  information  provided  in 
each  submenu  which  described  the  source  and  tone  of  each  elaboration.  Perhaps  textual  as  opposed  to  iconic  cues  to 
elaboration  type  allow  the  learner  to  more  easily  discern  the  link  destination,  thus  reducing  cognitive  effort”  (pp.  32- 
33). 

Leader  and  Klein  (1996  investigated  the  effects  of  search  tools  and  learner  cognitive  styles  on  performance 
in  searches  for  information  within  a hypermedia  database.  Results  indicated  that  field-independent  learners 
performed  significantly  better  than  field-dependent  learners  using  an  index/find  search  tool.  This  finding  led  to  a 
recommendation  that  designers  of  multimedia  materials  consider  learner  preferences.  They  also  suggest  further  study 
of  cognitive  style  factors  in  the  use  of  hypermedia  as  they  may  relate  to  the  formulation  of  criteria  for  interface  tool 
design. 

There  may  be  a difference  in  user  preference  for  font  sizes  in  Web-based  environments  depending  on  the 
specific  purpose  or  placement  of  the  text.  It  is  possible  that  preferences  may  differ  for  a body  of  text  as  opposed  to 
menu  text  on  a Web  page.  This  study  investigated  the  combined  elements  of  font  size  and  line  spacing  in  Web  menus 
for  both  a scrolled  and  not  scrolled  condition  with  a sample  of  undergraduate  university  students. 

Sample  and  Methodology 

The  subjects  were  185  university  students  enrolled  in  13  intact  sections  of  ET201,  Technology  in 
Education,  and  ET301,  Educational  Technology  Applications.  These  are  required,  undergraduate  preservice  teacher 
technology  courses  at  the  University  of  Northern  Colorado.  Participants  included  140  females  and  45  males.  All  but 
two  participants  were  native  English  speakers.  The  average  age  was  23,  with  a range  from  18  to  51.  Sixty  percent  of 
the  subjects  rated  their  Internet  use  as  at  least  six  hours  or  more  per  week;  only  five  subjects  responded  that  they 
spent  no  time  using  the  Internet.  A large  majority  of  the  subjects  (82%)  reported  that  they  were  comfortable  using  a 
computer. 

Course  instructors  (graduate  students  in  the  Educational  Technology  program)  were  asked  to  volunteer  their 
classes  for  this  study.  A total  of  13  classes  participated.  Four  different  researchers  administered  the  scripted 
treatment  over  a consecutive  five-day  period.  Students  were  asked  to  rank  their  preferences  of  four  distinct  Web 
menu  screens.  The  four  screens  included  these  combinations:  (a)  default  font,  double-spaced,  (b)  large  font,  double 
spaced,  (c)  default  font,  single  spaced,  and  (d)  large  font,  single  spaced.  Large  font  is  defined  as  HTML  “font=5.” 
These  screens  were  adapted  for  two  additional  conditions:  long  menus  (scrolled)  and  short  menus  (not  scrolled).  The 
short  menus  were  designed  to  be  fully  viewed  on  one  screen  without  scrolling. 

The  content  of  the  menus  was  selected  to  represent  a fairly  typical,  but  non-controversial,  subject  found  on 
the  Web;  lists  of  endangered  species  (see  Figure  1).  Every  screen  had  a title  section  in  large  type  that  said,  “Some 
Endangered  Species  in  North  America.”  Under  the  title  was  a screen  identification  in  a slightly  smaller  font  which 
simply  read  “Screen  A”  or  “Screen  B,”  etc.  A single  line  separated  the  title  area  from  the  menu  itself.  The  short 
menu  consisted  of  only  four  items  (e.g.,  American  Alligator,  Atlantic  Salt  Marsh  Snake)  under  the  heading, 
“Reptiles.”  The  longer  (scrolled)  menus  had  no  headings.  They  listed  a variety  of  endangered  species  in  no  particular 
order,  neither  alphabetical  nor  categorical.  The  32  items  in  the  scrolled  menus  were  of  varying  length,  from  single 
words  (e.g.,  Ocelot)  to  longer  names  (e.g.,  Carolina  Northern  Flying  Squirrel).  All  screen  backgrounds  were  white. 
The  menu  items  appeared  in  the  default  blue  color  of  the  browser.  All  Web  page  borders  were  the  default  (gray) 
color. 
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Figure  1.  EXAMPLE  NOT  SCROLLED  WEB  MENU 


Some  Endangered  Species  in  North  America 

Screen  B 


Reptiles 

• American  Alligator 

• American  Crocodile 

• Atlantic  Salt  Marsh  Snake 

• Plymouth  Red-Bellied  Turtle 


Data  were  collected  in  a computer  laboratory  setting  that  housed  twenty  7000  series  Power  Macintosh 
computers  with  monitors  set  to  standard  resolution  (640  x 480)  and  fifteen-inch  viewing  screens. 

The  ranking  task  was  given  to  each  class  during  the  first  10  minutes  of  the  class  session.  A script  of 
instructions  was  read  aloud  to  the  subjects;  then  the  ranking  sheet  and  demographic  data  collection  instrument  were 
distributed.  Due  to  concern  regarding  order  of  presentation  of  the  menu  combinations  potentially  influencing  user 
ranking,  four  URLs  were  created.  Each  URL  contained  four  screens  presenting  all  combinations  of  font  size  and  line 
spacing  in  either  the  scrolled  or  not  scrolled  condition.  Order  of  screen  presentation  varied  in  each  URL  for  both 
conditions  scrolled  and  not  scrolled.  Students  were  randomly  assigned  a URL  by  distributing  color-coded  ranking 
instruments  that  designated  a specific  URL  (set  of  four  menu  screens)  to  view.  The  students  were  instructed  to  open 
Netscape  Navigator  and  enter  the  URL  noted  on  their  ranking  sheet.  Subjects  were  given  five  minutes  to  view  the 
screens  and  rank  their  preferences  on  the  ranking  sheet.  At  the  end  of  five  minutes,  the  researcher  collected  the  sheets 
and  the  subjects  were  thanked  for  their  participation. 

Results 


Student  preferences  and  frequency  data  were  compiled  for  first,  second,  third,  and  least  preferred  menu 
styles  for  each  condition  (scrolled  and  not  scrolled).  The  results  reported  here  are  descriptive  in  nature  and  reflect  the 
characteristics  of  the  preservice  teacher  research  sample. 

The  first  research  question  asked  whether  font  size  or  line  spacing  would  affect  user  preference  in  a not 
scrolled  menu  list  displayed  as  a Web  menu.  Of  the  80  students  who  viewed  the  not  scrolled  menus,  60%  selected 
the  menu  with  a large  font  and  double  line  spacing  as  their  favorite,  33%  selected  a large  font  with  a single  spaced 
menu,  only  8%  preferred  the  default  font  size  whether  part  of  a single  spaced  menu  (5%)  or  a double  spaced  menu 
(3%).  Second  favorites  (ranked  number  two)  continued  the  preference  for  a large  font  size  with  28%  preferring  the 
large  font  size,  double  spaced  and  56%  preferring  the  large  font  size,  single  spaced.  Conversely,  the  least  preferred 
(ranked  number  four)  not  scrolled  menu  display  was  the  single  spaced,  default  font  size.  Eighty-five  percent  selected 
this  as  their  least  preferred  choice.  These  results  indicate  a strong  preference  for  a large  font  size  over  the  default 
font  size  for  Web  menus.  The  results  of  viewer  preferences  for  single  or  double  spacing  are  less  conclusive.  Refer  to 
Table  1 for  a summary  of  frequencies  and  percentages  of  student  rankings  of  the  not  scrolled  menu  options. 

The  second  research  question  investigated  whether  font  size  or  line  spacing  would  affect  user  preference  in 
a scrolled  menu  list  displayed  as  a Web  menu.  First  choice  preferences  of  the  105  students  who  viewed  the  scrolled 
menus  resulted  in  50%  who  selected  a menu  with  a large  font  and  double  line  spacing;  28%  selected  a large  font  with 
single  line  spacing;  only  12%  preferred  the  default  font  double  spaced  and  10%  preferred  the  default  font  single 
spaced.  For  students  who  ranked  the  scrolled  menus,  large  font  continued  to  be  the  most  preferred  choice  regardless 
of  spacing.  Forty-four  percent  of  the  students  selected  the  large  font  single  spaced  as  their  second  favorite  menu; 

21%  selected  the  large  font  double  spaced;  25%  selected  default  font  double  spaced;  and  only  10%  selected  the 
default  font  single  spaced.  The  fourth  preference  clearly  indicated  the  default  font  single  spaced  menu  was  the  least 
favorite  with  64%  ranking  this  choice  as  their  least  favorite.  Although  the  overall  preferences  identified  for  scrolled 
menus  reflected  the  same  patterns  of  choice  found  in  the  not  scrolled  menus,  the  preferences  were  less  pronounced 
than  for  the  not  scrolled  menus.  Refer  to  Table  2 for  a summary  of  frequencies  and  percentages  of  the  student 
rankings  for  the  scrolled  menu  options. 


Table  l 


FREQUENCIES  AND  PERCENTS  OF  PREFERENCES  IN  FONT  SIZE  AND  LINE  SPA  CING  IN  A NON 
SCROLLED  CRT  DISPLAYED  WEB  MENU 


Most  Second  Third  Fourth 

Favorite  Favorite  Favorite  Favorite 


Freq. 

% 

Freq. 

% 

Freq. 

% 

Freq. 

% 

Default  Font  Double  Spaced 

4 

5% 

12 

15% 

56 

70% 

8 

10% 

Large  Font  Double  Spaced 

48 

60% 

22 

28% 

7 

9% 

3 

4% 

Default  Font  Single  Spaced 

2 

3% 

1 

1% 

9 

11% 

68 

85% 

Large  Font  Single  Spaced 

26 

33% 

45 

56% 

8 

10% 

1 

1% 

Totals 

80 

101%* 

80 

100% 

80 

100 

80 

100% 

% 

* Rounding  error 

Table  2 

FREQUENCIES  AND  PERCENTS  OF  PREFERENCES  IN  FONT  SIZE  AND  LINE  SPACING  IN  A 
SCROLLED  CRT  DISPLA  YED  WEB  MENU 


Most 

Favorite 

Second 

Favorite 

Third 

Favorite 

Fourth 

Favorite 

Freq. 

% 

Freq.  % 

Freq 

% 

Freq. 

% 

Default  Font  Double  Spaced 

13 

12% 

26 

25% 

56 

53% 

10 

10% 

Large  Font  Double  Spaced 

52 

50% 

22 

21% 

14 

13% 

17 

16% 

Default  Font  Single  Spaced 

11 

10% 

11 

10% 

16 

15% 

67 

64% 

Large  Font  Single  Spaced 

29 

28% 

46 

44% 

19 

18% 

11 

10% 

Totals 

105 

100% 

105 

100% 

105 

99%* 

105 

100% 

♦Due  to  rounding 


Discussion 

This  study  investigated  font  size  and  line  spacing  for  Web  menus.  Results  clearly  indicate  that  users  prefer 
large  fonts  whether  presented  in  a scrolled  or  not  scrolled  Web  menu.  Double  line  spacing  was  preferred  over  single 
line  spacing  in  both  scrolled  and  not  scrolled  treatments.  The  combination  of  default  font  and  single  line  spacing  was 
overwhelmingly  ranked  as  least  preferred  in  both  the  scrolled  and  not  scrolled  conditions. 

A limitation  of  this  study  is  its  generalizability  to  alternative  settings.  The  subjects  of  this  study  were  a 
homogenous  group  of  undergraduate  students  with  the  shared  computer  experience  of  educational  technology 
classes.  Replication  of  this  study  with  subjects  outside  of  the  university  environment  is  recommended. 

Additionally,  the  reader  is  cautioned  to  note  that  these  data  reflect  user  preferences  and  may  or  may  not 
affect  the  users’  ability  to  learn  from  Web-based  environments  or  increase  their  Web  searching  efficiency. 

Additional  research  is  recommended  to  investigate  the  variables  of  font  size  and  line  spacing  in  Web  environments 
designed  for  instructional  purposes. 

Another  aspect  of  this  study  that  deserves  further  exploration  is  the  potential  connection  between  user 
preference  and  user  motivation.  Keller  and  Burkman  (1993)  state,  ‘The  rapidity  of  change  in  both  technical 
knowledge  and  cultural  perspectives  leads  to  increasing  demand  for  ongoing  education.  The  motivation  to  leam 
impacts  on  both  the  effectiveness  and  the  efficiency  of  instruction”  (p.  49).  With  the  increasing  emphasis  on  Web- 
based  systems  to  deliver  that  instruction,  it  is  imperative  that  instructional  designers  and  Web  developers  understand 
the  relationships  between  preferences,  motivation,  and  learning.  This  study  provides  a foundation  for  further 
exploration  of  these  relationships. 
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CBT  Design:  A Feedback  Aptitude  Treatment  Interaction 


Roy  B.  Clariana 

The  Pennsylvania  State  University 


Abstract 

A review  by  Clariana  (1993)  reported  an  aptitude-treatment  interaction  (ATI)  relating  instructional 
feedback  and  prior  knowledge.  For  low  prior  knowledge  learners , single-try  feedback  (STF)  was  slightly 
superior  to  multiple-try  feedback  (MTF),  while  for  high  prior  knowledge  learners,  MTF  was  superior  to 
STF.  In  this  present  report,  raw  score  data  from  existing  studies  that  examined  STF  and  MTF,  but  that  had 
not  considered  the  possibility  of  an  ATI  were  combined  and  reanalyzed  by  AN  OVA.  The  results  of  this 
analysis  are  presented  along  with  a tentative  explanation  of  the  observed  effect . Prescriptive  application 
of  the  ATI  and  suggestions  for  future  study  are  provided. 

Relevance 

Frequently  in  corporate  settings  and  occasionally  in  general  education  settings,  mastery  of  lower  level  skills 
are  often  critical  to  safety  or  performance.  For  example,  consider  the  multitude  of  alarm  sounds  that  can  occur  in  a 
nuclear  power  plant  and  the  correct  employee  reactions  to  these  alarms.  This  message  includes  verbal  information, 
lower  level  intellectual  skills,  and  possibly  some  attitudinal  objectives,  and  it  would  be  hard  to  dispute  that  this 
particular  message  is  not  important  to  the  employees  safety  and  also  their  feelings  of  well-being.  In  this  case, 
computer-based  training  (CBT)  can  accurately  produce  the  correct  sounds  and  can  require  the  employee  to  describe 
the  nature  of  the  emergency  and  the  correct  responses  to  each  category  of  emergency  (believe  it  or  not,  it’s  not 
always  run  away  as  fast  as  you  can).  This  type  of  CBT  module  would  be  easy  and  relatively  inexpensive  to  develop 
and  deliver  (especially  compared  to  instructor  led  training),  and  allows  for  new  employee  training  just-in-time  and 
refresher  training  for  existing  employees  as  needed  due  to  the  infrequency  of  such  alarms.  Decades  of  educational 
research  support  the  use  of  practice  with  feedback  for  learning  this  type  of  information.  This  presentation  argues  that 
learners'  prior  knowledge  should  be  considered  when  deciding  which  type  of  feedback  to  use  in  the  lesson  design. 

This  information  has  immediate  practical  application  for  CBT  designers. 

Note  that  this  is  not  new  data,  but  is  a reanalvsis  of  four  previous  experimental  studies  and  an  expansion  of 
an  earlier  review  by  Clariana  (1993).  Richey  (1998)  has  described  these  two  processes  as  an  accepted  form  of 
replicability,  though  caution  must  be  taken  that  this  present  report  is  not  seen  as  new  data.  This  reanalysis  and 
expansion  seem  acceptable  because  of  the  limited  circulation  of  that  earlier  review  due  to  the  demise  of  the  Journal 
of  Computer-based  Instruction  at  that  time  and  the  lack  of  information  regarding  this  possible  ATI  when  the  four 
experimental  studies  were  originally  conducted.  Further,  the  audience  for  this  presentation  are  the  most  potent  group 
for  utilizing  this  information. 

Introduction 

Advances  in  voice  recognition  technology  will  soon  allow  relatively  accurate  evaluation  of  spoken  words, 
and  neural  network  software  technology  may  eventually  allow  evaluation  of  sentences  and  paragraphs.  However, 
currently  computers  are  limited  in  the  types  of  learner  responses  that  can  be  evaluated  in  order  to  give  corrective 
feedback.  Computers  can  evaluate  learner  recognition  tasks  like  clicking  on  an  image  or  a word,  and  some  free 
recall  tasks,  such  as  typing  in  words  and  short  phrases. 

Feedback  in  these  circumstances  has  consistently  been  shown  to  be  better  than  no  feedback  (for  example, 
see  Clariana,  Ross,  & Morrison,  1991).  But  is  one  type  of  feedback  superior  to  another?  A meta-analysis  by 
Bangert-Drowns,  Kulik,  Kulik,  and  Morgan  (1991)  reported  effect  sizes  (ES)  that  compared  no  feedback  to  (a) 
right/wrong  feedback,  usually  referred  to  as  knowledge  of  response  feedback  (KR),  (b)  knowledge  of  correct 
response  feedback  (KCR),  (c)  try-again  feedback  (requires  multiple  tries),  and  (d)  elaborative  feedback.  Compared 
to  no  feedback,  KR  was  negatively  effective  (ES  = -0.08),  followed  next  by  KCR  feedback,  ES  = 0.22.  Repeat  until 
correct  and  elaborative  feedback  were  equally  most  effective  (ES  = 0.53).  The  performance  results  for  feedback  may 
be  summarized  as:  KR  < no  feedback  < KCR  < Repeat  until  correct  = Elaborative.  Prescriptively,  elaborative 
feedback  and  multiple-try  feedback  were  equally  most  effective,  however  elaborative  feedback  typically  is  more 
difficult  and  costly  to  design. 

A review  of  30  studies  by  Clariana  (1993)  took  this  a step  further  by  comparing  forms  of  multiple-try 
feedback  (MTF)  to  no  feedback,  KR  feedback,  KCR  feedback,  and  delayed  feedback.  The  median  effect  size  for 
MTF  compared  to  no  feedback  based  on  16  effect  size  measures  from  12  studies  was  ES  = 0.56.  This  finding  is 
nearly  identical  to  the  results  of  Bangert-Drowns  et.  al.  (1991)  above.  However,  unlike  that  meta-analysis,  the 
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results  for  MTF  compared  to  KR,  KCR,  and  delayed  feedback  were  all  mixed,  with  median  effect  sizes  close  to  no 
difference. 

To  explain  these  mixed  findings,  Clariana  (1993)  suggested  that  low-prior  knowledge  learners  benefited 
slightly  from  single-try  feedback  forms  relative  to  multiple-try  feedback  forms,  while  high-prior  knowledge  learners 
benefited  greatly  from  multiple-try  feedback  forms  relative  to  single-try  feedback  forms.  If  supported,  this  provides 
CBT  designers  with  clear  guidelines  for  selecting  number  of  feedback  tries  based  on  audience  analysis  data  and/or 
ongoing  lesson  performance. 

The  current  presentation  considers  this  feedback  by  prior  knowledge  aptitude  treatment  interaction  (ATI)  by 
combining  and  reanalyzing  data  from  the  existing  studies  described  in  Clariana  (1993).  This  study  addresses  the 
following  questions:  (a)  Is  there  an  interaction  between  learner  prior  knowledge  and  number  of  feedback  tries 
provided  by  CBT  lessons?  (b)  If  so,  how  large  is  this  effect? 


Approach 

A review  of  the  literature  was  conducted  to  identify  all  available  studies  that  compared  MTF  and  STF  while 
also  providing  raw  score  data.  It  was  necessary  that  besides  posttest  measures,  the  data  set  must  also  contain  some 
measure  of  prior  knowledge  of  the  content,  so  that  individual  subjects  could  be  classified  as  "low"  or  "high"  in  terms 
of  prior  knowledge.  In  all  cases,  lesson  data  ultimately  was  used  to  form  low  and  high  prior  knowledge  groupings 
based  on  median  split  of  lesson  scores. 


» Three  dissertations  and  one  published  report  were  obtained  that  met  these  criteria.  All  four  utilized  computer-based  delivery 
of  the  instructional  materials.  The  studies  included:  Noonan  (1984)  n = 60,  Clariana  and  Smith  (1989)  n = 48,  Nielson 
(1990)  n = 173,  and  Clariana  ( 1990)  n = 40. 

■ Posttest  raw  scores  were  converted  to  standard  scores  within  each  study,  then  these  standard  scores  were  combined  into  one 
data  set  and  analyzed  by  2x2x4  analysis  of  variance  (ANOVA)  with  the  factors  feedback  (MTF  and  STF),  prior  knowledge 
(low  and  high),  and  study  (the  four  studies  listed  above).  The  Final  sample  contained  n = 321  subjects. 

The  main  effect  for  number  of  feedback  was  not  significant  (see  Table  1).  The  main  effect  for  prior 
knowledge  was  highly  significant  F(l,  305)  = 24.21,  p=.0001.  As  would  be  expected,  those  students  that  performed 
well  on  the  CBT  lesson  also  performed  better  on  the  posttest.  This  finding  is  interesting  in  that  in  experimental 
studies  using  grouping  by  lesson  performance,  there  is  typically  a regression  to  the  mean.  The  “low”  group  tends  to 
gain  quite  a bit  since  their  lesson  scores  are  low,  while  the  “high”  group  gains  less  and  may  even  decline.  If  the 
experimental  treatment  is  not  strong  enough,  this  regression  to  the  mean  causes  a leveling  effect  that  can  wash-out 
low  and  high  group  differences. 


Table  1.  Posttest  scores  Analysis  of  Variance 


Source 

df 

SS 

MS 

F 

E 

Feedback  tries  (T) 

1 

0.14 

0.14 

0.16 

0.69 

Prior  knowledge  (K) 

1 

22.02 

22.02 

24.21 

0.00* 

FxT 

1 

5.05 

5.05 

5.55 

0.02  * 

Study  (S) 

3 

0.06 

0.02 

0.02 

0.99 

TxS 

3 

2.15 

0.72 

0.79 

0.50 

KxS 

3 

0.45 

0.15 

0.16 

0.92 

TxKxS 

3 

1.96 

0.65 

0.72 

0.54 

Error 

305 

277.34 

0.91 

A significant  interaction  of  feedback  and  prior  knowledge  was  observed  F(l,  305)  = 5.55,  p=.02.  For  low 
prior  knowledge  students,  single-try  feedback  was  more  effective  than  multiple-try  feedback,  ES  = 0.1 1 (see  Figure 
1).  For  high  prior  knowledge  learners,  multiple-try  feedback  was  better,  ES  = 0.39. 
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Figure  1.  Interaction  of  prior- know  ledge  and  type  of feedback. 


Prior  Knowledge 

The  low-prior  knowledge  students  made  large  lesson  (L)  to  posttest  (P)  gains  with  both  single-try  and 
multiple-try  feedback  (see  Figure  2).  The  high-prior  knowledge  students,  since  they  had  already  scored  high  on  the 
lesson  measure  of  content  knowledge,  did  not  make  gains  as  great  as  the  low-prior  knowledge  students.  In  addition, 
the  high  prior-knowledge  students  receiving  multiple-try  feedback  (the  right  vector  of  the  “high”  pair  of  vectors) 
made  larger  gains  on  the  posttest  relative  to  the  high-prior  students  receiving  single-try  feedback  (the  left  vector  of 
the  “high”  pair  of  vectors)  even  though  both  groups  already  had  good  content  knowledge. 


Figure  2.  Lesson  score  (L)  to  Posttest  ( P)  gains  as  vectors  for  single-try  ( left)  and  multiple-try  ( right)  at  low 


low  high 


Discussion 


prior  knowledge 


To  summarize,  this  reanalysis  supports  an  ATI  for  prior  knowledge  and  number  of  feedback  tries.  Our 
tentative  explanation  of  this  effect  involves  an  attitudinal  impact  on  learning.  Note  that  for  these  four  studies,  and 
for  most  if  not  all  MTF  studies  so  far,  there  is  no  difference  in  presentation  form  between  STF  and  MTF  when  the 
learner's  response  is  correct.  The  difference  in  presentation  form  between  STF  and  MTF  occurs  only  with  lesson 
errors,  in  that  the  learner  must  respond  again  with  MTF  but  is  usually  provided  the  correct  answer  without  additional 
responses  with  STF.  Dick  and  Latta  (1970)  reported  that  low  ability  students  became  frustrated  with  repeat  until 
correct  (MTF)  recall  lessons  relative  to  high  able  students  and  also  the  STF  treatment.  To  examine  possible 
frustration  with  MTF,  we  recently  collected  item-by-item  posttest  responses  for  twenty  of  our  students  who  were 
taking  a computer-based  module  as  an  assignment  in  an  instructional  design  class.  In  that  module,  it  was  possible  to 
intentionally  skip  posttest  questions  (recall  level)  by  entering  either  nothing  or  else  entering  nonsense  letters 
(“garbage”)  and  pressing  Enter.  Figure  3 below  shows  the  total  number  of  intentionally  skipped  posttest  items  for 
the  STF  group  (n=10)  and  MTF  group  (n=10).  Note  that  the  posttest  was  identical  for  each  treatment,  thus  the 
pattern  observed  relates  to  attitude  carried  over  from  the  preceding  lesson  activity. 
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Figure  3 . Cumulative  posttest  “skip”  or  “garbage"  responses. 


posttest  question  number 

This  data  leads  us  to  feel  that,  as  in  the  Dick  and  Latta  (1970)  study,  students  may  experience  some 
frustration  with  MTF,  and  this  will  likely  negatively  impact  learning. 

At  the  same  time,  in  a survey  of  the  same  class,  71%  of  the  students  said  that  they  would  prefer  to  try-again 
(MTF)  if  given  the  choice.  Some  specific  comments  included:  "I  learn  more  when  I get  it  on  my  own"  and  "Trying  a 
second  time  makes  me  think  more  about  the  question  and  what  it  is  really  asking".  This  suggests  that  these  learners 
and  maybe  learners  in  general  have  a bias  towards  "getting  it  right  on  their  own",  and  so  if  they  don't  miss  so  many 
items  that  it  becomes  punishing,  then  they  actively  engage  the  items  and  learn  more  (relative),  thus  accounting  for  the 
higher  performance  for  MTF  with  high  prior  knowledge  learners.  But  if  the  learner  misses  a number  of  items,  they 
begin  to  become  frustrated  and  may  eventually  stop  trying  to  learn,  thus  the  lower  performance  for  MTF  (relative) 
for  the  low  prior  knowledge  learners.  Providing  the  correct  answer  after  one  try  (STF)  tends  to  reduce  frustration  for 
the  students  who  miss  a lot  of  items,  but  it  seems  to  come  at  the  cost  of  reduced  engagement  for  the  students  who  are 
doing  well.  Thus,  the  type  of  feedback  given  can  influence  the  learners  approach  to  the  lesson,  and  on-going  lesson 
performance  serves  as  advisement  allowing  the  student  to  decide  to  engage  in  or  disengage  from  the  lesson. 

Requiring  high-prior  knowledge  learners  to  try  again  in  CBT  has  intuitive  appeal,  and  can  be  implemented 
by  tracking  lesson  performance  and  basing  feedback  prescription  of  STF  or  MTF  on  that  performance.  A lesson  may 
start  out  with  MTF,  if  a students  on-going  lesson  performance  drops  below  some  designated  value,  say  80%,  then  the 
lesson  changes  to  STF. 

Does  this  finding  generalize  to  other,  and  non-CBT  settings?  For  example,  in  a constructivist-based 
classroom,  what  are  the  ramifications  of  asking  questions  to  high-prior  knowledge  students  versus  the  low-prior 
knowledge  students.  Asking  a difficult  question  and  requiring  a student  to  develop  the  answer  parallels  the  MTF 
condition  with  low  prior  knowledge  students.  This  approach  may  be  frustrating  to  the  student,  and  could  result  in 
disengagement  and  reduced  learning.  Of  course,  teachers  in  class  have  additional  options  to  mitigate  frustration,  but 
the  question  remains  about  what  is  the  best  approach  in  terms  of  questioning  level  and  which  students  should  be 
asked  to  respond.  If  this  ATI  holds  for  classroom  instruction,  in-class  questions  should  be  carefully  crafted  and 
targeted  to  the  individual  student,  along  the  lines  of  Vygotsky’s  zone  of  proximal  development  (Vygotsky,  1986),  and 
maybe  asking  some  "easy"  questions  is  a good  idea. 

Though  this  ATI  is  only  supported  by  four  experimental  studies,  there  is  enough  evidence  to  warrant  future 
studies  to  confirm  or  deny  it,  and  possibly  to  expand  on  these  findings  to  other  settings.  Though  it  is  not  currently 
fashionable  to  investigate  "lower  level"  domains  such  as  verbal  learning  using  simple  (or  not  so  simple)  recognition 
tasks  like  answering  multiple-choice  tests,  nevertheless,  these  domains  are  an  important  and  foundational  to  learning; 
and  multiple-choice  items,  especially  with  feedback,  are  one  tool  among  many  that  any  instruction  designer  may  use. 
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Abstract 

While  there  have  been  studies  investigating  the  effect  of  instructional  interventions , few  systematic 
principles  have  been  identified  for  providing  support  for  learners  who  are  poor  self  regulators.  We 
analyzed  the  growing  body  of  research  on  self  regulation  and  the  distinctive  self  regulation  differences 
between  higher  and  lower  achieving  adult  learners  to  identify  four  instructional  principles  to  infuse 
compensatory  support  for  weak  self  regulation.  A rationale  based  on  research  supports  each  principle  and 
instructional  application  examples  are  included. 

Introduction 

A plethora  of  research  describes  how  learner  individual  differences  influence  the  effectiveness  of  instruction 
although  only  recently  have  instructional  designers  proposed  seminal  instructional  guidelines  for  addressing  learner 
differences  to  improve  learning  outcomes  (Jonassen  & Grabowski,  1993).  Even  more  recently  instructional  designers 
began  investigating  the  role  of  self  regulation  (SR),  an  important  individual  difference,  in  instruction  (Young,  1996). 
Self-regulation  determines  how  students  personally  activate,  alter,  and  sustain  their  learning  (Zimmerman  & 
Martinez-Pons,  1986).  Students  are  self-regulated  to  the  degree  that  they  are  metacognitively,  motivationally,  and 
behaviorally  active  participants  in  their  own  learning  process  (Zimmerman  & Martinez-Pons,  1986).  Students  who 
engage  in  self-regulation  take  responsibility  for  and  initiate  their  own  efforts  to  acquire  skill  and  knowledge  instead 
of  depending  upon  external  sources  (Zimmerman  Martinez-Pons,  1988).  Students  who  self  regulate  learning  have 
accessed  metacognitive  strategies,  were  more  aware  of  their  limitations,  and  have  taken  steps  to  correct  deficiencies 
(Zimmerman  & Martinez-Pons,  1995).  Self  regulated  learners  continually  plan,  organize,  monitor,  and  evaluate  their 
learning  processes  (Como,  1989;  Hagen  & Weinstein,  1995;  Zimmerman  & Paulsen,  1995). 

SR  has  been  consistently  related  to  achievement  in  learners  across  age  groups.  Ninety-three  percent  of  80 
high  school  participants  were  correctly  classified  into  preassigned  high/low  achievement  tracks  based  on  strategy  use 
indicators  (Zimmerman  & Martinez-Pons,  1986).  In  a subsequent  study  with  participants  from  fifth,  eighth,  and 
eleventh  grades,  students  who  were  classified  as  gifted  reported  significantly  greater  use  of  self-regulated  learning 
strategies  than  did  students  classified  as  regular  achieving  (Zimmerman  & Martinez-Pons,  1990).  Six  SR  strategies 
correlated  positively  with  college  statistics  student’s  exam  scores:  self  evaluation,  seeking  information,  rehearsal  and 
memorization,  reviewing  texts,  and  reviewing  tests  (Lan,  1998). 

Fortunately,  SR  is  not  inherent  but  is  a learned  response  that  can  be  taught  and  controlled  by  the  student 
(Iran-Nejad,  1990).  Furthermore,  even  though  a student  is  already  exhibiting  self-regulation,  these  processes  can  be 
enhanced  to  better  support  learning,  motivation,  and  performance  (Pintrich,  1995).  Preservice  teachers  enrolled  in 
education  courses  have  benefited  from  instruction  that  supported  self-regulated  learning  strategies  such  as  cognitive 
skill  instruction,  effort  reinforcement,  and  metacognitive  skill  use  (Schutz,  Lanehart,  & White,  1995).  Metacognitive 
strategy  instruction  when  embedded  in  content  instruction  has  improved  student  achievement  (Nist,  Simpson, 

Olejnik,  & Mealey,  1991).  Structuring  regulatory  strategies  within  instruction  has  improved  achievement  in  college 
statistics  students  (Lan,  1998).  The  evidence  from  intervention  studies  on  one  or  more  of  the  SR  processes  suggests 
that  instruction  that  supports  regulation  improves  learning  outcomes. 

Better  learners  probably  employ  some  of  the  strategies  associated  with  SR  to  compensate  for  cognitive 
deficiencies  (Di  Vesta  & Moreno,  1993).  Given  that  SR  activities  may  be  increased  through  instruction,  poor  SR  is  a 
set  of  learner  activities  for  which  instruction  may  be  able  to  compensate.  Compensating  regulatory  instruction  may 
moderate  the  negative  impact  of  poor  SR  on  achievement.  For  learners  who  are  less  self  regulated,  the  instructional 
environment  may  compensate  for  poor  SR  skills  with  external  regulatory  instructional  interventions.  Instruction  that 
includes  lower  achieving  learners  in  the  target  population  may  be  able  to  improve  learning  outcomes  if  the  design 
promotes  compensatory  SR  strategies. 

While  there  have  been  studies  investigating  the  effect  of  SR  instructional  interventions  (Hattie,  Biggs,  & 
Purdie,1996;  Lan,  1998;  Young,  1996),  and  SR  models  for  instruction  (Schunk  & Zimmerman,  1998)  and  cognitive 
models  of  SR  based  upon  the  expert,  highly  self  regulating  learner  (c.f.,  Winne,  1995),  few  systematic  principles 
have  been  identified  for  providing  support  for  poor  self  regulators  based  upon  their  SR  weaknesses.  Some  SR 
interventions  have  been  content  specific,  such  as  writing  (Graham,  Harris,  & Troia,  1998),  reading  comprehension 
(Pressley,  El-Dinary,  Wharton-McDonald,  & Brown,  1998),  or  mathematics  (Schunk,  1998).  Others  have  included 
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effective  college  learning  to  learn  courses  (Hofer,  Yu,  & Pintrich,  1998)  or  have  described  “computer  supported 
tools”  to  guide  learner  SR  (Winne  & Stockley,  1998,  p.133).  Other  approaches  are  directed  toward  specific 
populations  such  as  adolescents  (Belfiore  & Homyak,1998),  learning  disabled  (Butler,  1998),  and  children 
(Biemilier,  Shany,  Inglis,  & Michenbaum,1998;  c.f.  , Como,  1995).  Some  SR  instructional  models  have  approached 
the  problem  from  the  teacher’s  but  not  the  designer’s  perspective  (c.f.,  Schunk  & Zimmerman,  1998);  in  fact,  “most 
existing  applied  research  on  SRL  has  identified  ways  teachers  can  improve  or  promote  SRL  (Como,  1995,  p.  201).” 

Few  SR  principles  or  models  have  been  based  upon  empirically  supported  SR  needs,  that  is,  discrepant  SR 
activities  associated  with  achievement.  Evidence  suggests  that  the  closer  the  match  between  learner  SR  needs  and  the 
instructional  environment,  the  more  positive  the  learning  results  (Young,  1996;  Ertmer,  Newby,  & MacDougall, 
1996).  A meta-analysis  analysis  with  study  participants  across  K-16  indicated  that  learning  has  resulted  from 
learning  skill  interventions  although  improved  outcomes  were  less  likely  when  the  intervention  targeted  a selected 
deficit  rather  than  multiple  deficits  (Hattie,  Biggs,  & Purdie,  1996).  Furthermore,  because  SR  may  be  a 
developmental  process  which  takes  years  and  aging  and  even  then  may  not  transfer  well  (Pressley,  1995),  the  more 
that  instruction  can  prompt  or  foment  regulation,  the  more  the  poorly  self  regulated  learner  may  learn.  Instruction 
that  follows  principles  for  SR  may  bridge  the  gap  between  unacceptable  and  sufficient  learning  outcomes.  Indeed 
design  principles  may  be  indispensable  to  instructional  environments  for  the  learner  without  SR  expertise. 

Instructional  principles  that  guide  the  coordination  between  learner  needs  and  the  instructional  environment 
might  result  in  more  effective  instruction  although  the  instructional  design  literature  lacks  general  instructional 
design  principles  to  facilitate  the  match  between  SR  needs  and  instruction.  We  propose  four  instructional  design 
principles,  independent  of  content  or  a delivery  media,  to  address  discrepant  SR  activities  associated  with 
achievement.  We  have  drawn  upon  two  bodies  of  literature  to  derive  the  four  SR  principles.  The  first  source  is  the  a 
large  body  of  research  encompassing  SR  and  the  many  individual  factors  that  affect  or  comprise  SR,  e.g., 
achievement  levels,  metacognition,  academic  time  management.  The  second  source,  research  on  SR  characteristics 
associated  with  achievement,  included  three  empirical  studies:  the  first  validated  an  SR  interview  measure  that 
classified  learners  as  higher  or  lower  achieving  (Zimmerman  & Martinez-Pons,  1995);  the  second  classified  adult 
learners  into  normal  and  lower  achieving  with  the  same  SR  interview  measure  (Ley  & Young,  1998);  and  a third 
analyzed  SR  differences  between  adult  learners  who  used  a SR  instructional  protocol  and  those  who  did  not  (Lan, 
1998).  We  first  analyzed  research  on  learner  behaviors  associated  with  SR.  We  then  analyzed  the  significant  SR 
differences  between  higher  and  lower  achieving  learners.  Finally  we  generated  principles  derived  from  the  research 
on  SR  influences  to  address  the  significant  SR  differences  associated  with  achievement. 

Salient  SR  Activities 

Two  studies  (Lan,  1998;  Ley  & Young,  1998)  have  classified  discrepant  SR  activities  among  college 
learners  according  to  categories  and  measures  developed  through  the  research  of  Zimmerman  and  Martinez-Pons 
(1986)  (See  Table  1).  Each  study  measures  SR  in  the  same  categories  although  one  was  a Likert  scale  and  one  was 
an  interview.  Since  SR  data  collected  with  Likert  scales  lack  validity  that  interview  measures  overcome  with  lower 
achieving  adults  the  interview  was  used  to  collect  data  (Ley  & Young,  1998;  Young  & Ley,  1997).  Based  upon 
structured  interviews,  five  categories  of  strategy  use  were  significantly  different  between  normal  and  lower  achieving 
adults  (Ley  & Young,  1998).  The  five  categories  were,  in  order  from  the  strongest  relationship,  self  evaluation, 
environmental  structuring,  organizing  and  transforming,  reviewing  tests,  and  keeping  records  and  monitoring. 

Three  of  the  SR  strategies  that  differentiated  between  normal  and  lower  achieving  adults  (Ley  & Young, 
1998)  were  also  significantly  different  between  adult  learners  who  did  and  did  not  use  an  instructional  self 
monitoring  protocol  (Lan,  1996):  environmental  structuring,  reviewing  tests,  and  self  evaluation.  The  significant  SR 
differences  between  learners  who  were  required  to  use  an  instructional  self  monitoring  protocol  and  those  who  did 
not,  in  order  from  the  strongest  relationship,  were  environmental  structuring,  reviewing  tests,  self  evaluation,  and 
rehearsing  and  memorizing.  Using  the  self  monitoring  protocol  was  also  associated  with  significantly  higher  exam 
scores.  In  both  studies,  seeking  assistance  with  peers  significantly  differed  but  the  relationship  was  negative;  peer 
assistance  seekers  were  more  likely  to  be  either  the  lower  achieving  learners  (Ley  & Young)  or  not  to  have  used  the 
self  monitoring  protocol  (Lan). 
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Table  1.  Self  Regulation  Categories  (from  Zimmerman  & Pons,  1986) 

1.  Self  evaluation  (1,3)-  student  initiated  evaluations  on  the  quality  of  completed  work. 

2.  Organizing  and  transforming  (3,0)-  overt  or  covert  rearrangement  of  instructional  materials  to  improve  learning,  e.  g.,  outlining  before 
writing  a paper 

3.  Goal-setting  and  planning  - student-identified  desirable  educational  outcomes  or  subgoals  and  planning  for  sequencing,  timing,  and 
completing  activities  related  to  goals 

4.  Seeking  information  - student-initiated  efforts  to  secure  further  task  information  from  nonsocial  sources  when  undertaking  an  assignment, 
e.  g.,  searching  the  internet 

5.  Keeping  records  & monitoring  (5,  0)-  student-initiated  efforts  to  record  events  or  results. 

6.  Environmental  structuring  (2,  1)  - student-initiated  efforts  to  select  or  arrange  the  physical  setting  to  make  learning  easier 

7.  Self  consequences  - student  arrangement  of  rewards  or  punishment  for  success  or  failure 

8.  Rehearsing  & memorization  (0,  4)  - student-initiated  efforts  to  memorize  material  to  overt  or  covert  practice 

9.  Seeking  assistance  from  experts/teachers  - student-initiated  efforts  to  solicit  help  from  experts  or  faculty 

10.  Seeking  assistance  from  peers1  - student -initiated  efforts  to  solicit  help  from  other  learners. 

1 1 . Reviewing  tests  (4,  2)  - student-initiated  efforts  to  reread  tests  to  prepare  for  class  or  further  testing 

12.  Reviewing  notes  - student-initiated  efforts  to  reread  notes  to  prepare  for  class  or  further  testing 

13.  Reviewing  texts  - student-initiated  efforts  to  reread  textbooks  to  prepare  for  class  or  further  testing 

Note,  The  first  and  second  number  in  parentheses  represent  the  rank  by  strength  of  significant  relationships,  respectively,  between  lower  and 
normal  achieving  adults  (Ley  & Young,  1998)  and  adult  users  and  nonusers  of  a self  monitoring  instructional  protocol  the  (Lan,  1996). 

1 This  was  the  only  significant  negative  relationship  between  lower  and  normal  achieving  adults  (Ley  & Young,  1998)  and  adult  users  and 
nonusers  of  a self  monitoring  instructional  protocol  (Lan,  1996) 


Compensatory  SR  Principles 

We  propose  four  instructional  design  principles  to  infuse  compensatory  instructional  support  for  adult  SR 
deficiencies.  Instruction  that  accounts  for  SR  may  be  essential  for  effective  instruction  with  low  achieving,  under 
prepared,  or  ineffective  learners.  Both  more  and  less  self  regulated  learners  should  benefit  from  the  regulatory 
support  the  principles  prescribe  by  embedding  SR  strategies  in  instruction  to  compensate  for  SR  deficiencies.  For  the 
less  self  regulated  learner,  the  instruction  would  embed  information  and  activities  to  compensate  for  SR  weaknesses; 
the  same  information  and  activities  should  further  encourage  SR  among  more  skillful  learners. 

More  self  regulating  adult  learners  may  benefit  from  a reduction  of  the  cognitive  load  SR  imposes  thereby 
allowing  the  learner  to  spend  less  effort  on  SR  activities.  Many  adult  students  often  have  job  and  family 
responsibilities  that  compete  with  educational  commitments  for  their  time  and  energy;  course-embedded  SR 
guidance  gives  the  learner  SR  tools  and  strategies  that  he  or  she  may  otherwise  have  to  take  time  to  construct  or 
identify.  Learner  may  spend  time  deploying  the  SR  strategies  instead  of  creating  them. 

We  present  four  principles,  each  with  a rationale,  and  one  for  each  of  four  broad  categories  of  SR  activities: 
preparing,  organizing,  monitoring,  and  evaluating.  The  definitions  for  each  of  the  processes  were  adapted  from  the 
work  of  Zimmerman  and  Martinez-Pons  (1986)  (See  Table  1).  The  instructional  principles  are  adaptable  across 
media  so  that,  for  example,  an  instructor,  CAI,  or  web  site  could  be  the  instructional  media  for  providing 
compensatory  SR  activities. 


Principle  1:  Prepare 

The  instruction  should  encourage  and  guide  learners  to  prepare  appropriate  learning  environments. 
Rationale 

Environmental  structuring,  that  is  selecting  and  arranging  the  physical  setting  to  make  learning  easier,  has 
been  strongly  associated  with  SR.  Arranging  the  physical  setting  was  the  second  strongest  predictor  achievement 
levels  between  normal  achieving  and  lower  achieving,  under  prepared  college  students  (Ley  & Young,  1998);  it  was 
the  strongest  difference  between  college  students  who  used  an  instructional  self  monitoring  protocol  and  those  who 
did  not  (Lan,  1996).  The  learners  who  recorded  their  study  behaviors  with  the  protocol  reported  significantly  greater 
environmental  structuring  and  earned  significantly  higher  exam  scores  than  those  who  recorded  instructor  behaviors 
or  did  no  recording.  Structuring  the  environment  includes  the  learner’s  ability  to  cope  effectively  with  disturbances,  a 
crucial  part  of  self  regulation  and  volition  (Como,  1994). 

Preparing  Strategies 

Include  as  part  of  the  instruction  techniques  and  exercises  for  establishing  a study  area  that  is  quiet, 
comfortable,  without  distractions;  encourage  learners  to  analyze  how  and  when  to  effectively  control  distractions  (c. 
f.,  Como,  1994).  Suggest  that  students  find  a quiet  place  to  work  and  eliminate  distractions  such  as  phone  calls  and 
noise.  Suggestions  to  eliminate  explicitly  identified  distractions  could  be  included  with  a first  assignment.  For 
example,  in  a distance  course,  students  who  do  poorly  on  the  first  assignment  could  receive  counseling  on  structuring 
their  study  environment  (c.  f.,  Como). 


Instruction  should  require  students  to  establish  a study  environment  with  the  characteristics  that  facilitate 
concentration  and  attention.  For  example,  the  instruction  could  include  a checklist  that  would  require  students  to 
confirm  they  have  established  an  appropriate  study  environment  that  is  free  from  distractions  such  as  television  and 
noise.  Instruction  could  require  learners  to  record  the  amount  of  time  spent  studying  in  this  environment  and  submit 
the  record  as  a learning  requirement.  This  strategy  encourages  the  student  to  reflect  upon  how  prepared  he  or  she  is 
to  study  and  encourages  the  learner  to  prepare  the  environment  in  which  he  or  she  will  be  able  to  study  most 
effectively. 

Principle  2:  Organize 

The  instruction  should  include  different  arrangements  of  the  learning  materials  that  will  facilitate  learning 
and  the  learner’s  cognitive  and  metacognitive  strategy  use. 

Rationale 

Organizing  and  transforming  activities  were  the  third  strongest  predictor  of  achievement  level  between 
normal  and  lower  achieving  college  students  (Ley  & Young,  1998).  Matching  appropriate  strategies  to  outcomes  has 
improved  learning  outcomes  (McKeague  & Di  Vesta,  1996).  Learners  have  benefited  from  meaningful  and  directed 
practice  (Karabenick  & Knapp,  1990)  and  from  using  cognitive  and  metacognitive  strategies  (Brozo,  Stahl,  & King, 
1996;  Davis,  1994;  Frazier,  1993;  Nist  & Simpson,  1990).  Learners  tend  to  use  study  strategies  because  they  are 
familiar  not  because  they  are  effective  (Gamer,  1990)  therefore  the  instruction  can  extend  the  range  of  strategies  by 
structuring  how  to  learn  the  targeted  skill  or  concepts.  Because  to  some  extent  SR  is  a content  dependent  activity 
(Alexander,  1995),  including  proven  content  specific  learning  approaches  should  strengthen  the  instruction  by 
presenting  effective  ways  to  learn  a particular  domain. 

Organizing  strategies 

The  instruction  might  suggest  or  require  the  learner  to  use  a specified  order  and  approach  for  learning 
concepts  and  skills  taught  in  a module.  Study  techniques  for  learning  particular  skills,  such  as  mathematics  or 
reading  comprehension,  may  be  included  with  examples  or  demonstrations  of  cognitive  and  metacognitive  strategies 
(c.  f.,  Hofer,  Yu,  & Pintrich,  1998).  The  instruction  could  include  content  outlines  or  advance  organizers  that  will 
structure  the  learning  sequence  and  identify  concepts  for  individual  modules  and,  with  less  detail,  for  a course.  Unit 
glossaries  or  graphic  organizers  could  define  and  present  relationships  among  concepts  and  skills  presented  in  the 
instruction. 

Principle  3:  Monitor 

Prompt  the  learner  to  record  how  and  when  he  or  she  engages  in  learning  and  the  results  of  the  effort. 
Rationale 

“Self  monitoring  is  an  important  component  of  self  regulated  learning  . . . and  faculty  can  help  students 
learn  how  to  self  monitor”  (Zimmerman  & Paulsen,  1995,  p.  13).  Self  regulators  use  external  and  internal  feedback 
to  monitor  how  well  they  are  meeting  learning  goals,  how  effective  their  learning  strategies  and  tactics  are,  and  the 
quality  of  their  learning  outcomes  (Butler  & Winne,  1995).  Feedback  “is  inherent  in  and  a prime  determiner  of 
processes  that  constitute  SR  learning”  (Butler  & Winne,  p.  245).  Learners  who  exercise  metacognition  are  aware  of 
their  own  learning  processes,  the  results  of  the  processes,  and  can  adjust  their  behavior  or  thinking  to  improve  or 
correct  deficient  learning  processes  (Gagne  & Glaser,  1987).  They  monitor  their  progress  by  using  metacognitive 
strategies  to  adjust  cognitive  strategies  that  compensate  for  progress  or  difficulties  (Gamer,  1990;  Pressley, 
Woloshyn,  Lysynchuk,  Martin,  Wood,  & Willoughby,  1990;  Ridley,  Schutz,  Glanz,  & Weinstein,  1992).  Learners 
who  recorded  their  learning  activities  on  a self  monitoring  protocol  scored  higher  on  examinations  than  did  learners 
who  recorded  teacher  activities  or  did  not  record  either  (Lan,  1996). 

Monitoring  strategies 

Instruction  may  include  a form  for  monitoring  learning  activities  and  require  the  learner  to  log  her  time 
spent  on  learning  activities  such  as  taking  notes  in  class,  reading  the  text,  attending  a lecture,  working  homework 
problems,  etc.  (c.  f.,  Lan,  1998).  The  instruction  may  require  the  learner  to  submit  his  progress  tracking  as  part  of  the 
instructional  process.  The  instruction  should  present  explicit  interim  process  and  product  assignments  that  are  the 
occasion  for  feedback  and  that  lead  the  student  to  the  desired  learning  goals. 

The  instructional  process  should  offer  the  learner  frequent  feedback  on  interim  process  and  product 
assignments  that  includes  whether  his  or  her  response 'is  correct  or  not,  why  it  is  incorrect,  and  how  to  correct  the 
response.  The  instruction  should  allow  for  the  external  feedback  on  interim  process  and  products  that  the  student 
submits  on  the  way  to  the  terminal  learning  outcome.  For  example,  if  the  learning  goal  is  to  write  a technical  report, 
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the  instruction  may  require  the  learner  to  submit  and  receive  feedback  first  on  the  quality  of  the  learner’s  proposed 
information  sources,  then  an  outline  of  the  report,  followed  by  feedback  on  a draft  of  the  report,  graphics  that  support 
the  report,  etc.  In  other  words  the  instruction  incorporates  learner  performances  and  learner  feedback  on  many,  if  not 
most,  of  the  component  activities  in  the  order  in  which  they  are  accomplished  to  produce  the  desired  learning  goal. 

Instruction  might  deliver  to  learners  frequent  printed  progress  reports  that  compare  what  they  have  learned 
or  accomplished  to  date  to  what  future  assignments  will  be  required  to  meet  the  instructional  goal.  Feedback  should 
indicate  progress  toward  learning  goals,  that  is,  mastery  and  what  the  learner  should  know  or  be  able  to  do,  and  not 
toward  performance  goals,  that  is,  scores,  grades,  relative  standing  since  a wealth  of  research  indicates  the  former  are 
associated  with  higher  achievement  and  the  latter  are  counterproductive  (Hagen  & Weinstein,  1995).  Therefore 
Feedback  which  includes  scores  should  reflect  degree  of  mastery  not  relative  standing  in  a group  of  learners. 

Principle  4:  Evaluate 

Instruction  should  enable  learners  to  determine  how  well  they  have  learned.  Self  evaluation  occurs  when  the 
student  reflects  upon  the  quality  of  his  or  her  completed  work  to  prepare  for  further  learning  or  performance 
evaluations.  Similar  to  monitoring,  self  evaluation  entails  learner  comparisons  between  what  he  or  she  has 
accomplished  to  a standard  or  goai. 

Rationale 

Self  evaluation  processes  have  been  significantly  weaker  in  low  achieving  high  school  students 
(Zimmerman  & Pons,  1990)  and  college  students  (Lan,  1996;  Ley  & Young,  1998)  when  compared  to  their  higher 
achieving  counterparts.  Furthermore,  higher  achieving  students  review  previous  tests  significantly  more  than  do 
lower  achieving  college  students  (Lan,  1996;  Ley  & Young,  1998).  For  instructional  purposes,  reviewing  tests  serves 
the  same  function  as  self  evaluation  because  both  induce  the  student  to  reflect  upon  instances  in  which  the  quality  of 
his  or  her  learning  fell  short  of  the  quality  required.  College  statistics  students  who  recorded  their  study  behaviors 
reported  reviewing  previous  tests  significantly  more  often  than  did  those  who  recorded  instructor  behaviors  or  did  no 
recording  (Lan,  1998).  Less  regulated  learners  do  not  attend  to  their  own  course  progress  with  the  same  diligence  as 
the  more  self  regulated  learners. 

Evaluating  strategies 

Instruction  should  require  learners  to  submit  and  receive  feedback  on  interim  learning  processes  and 
products.  For  example,  computer  based  instruction  which  tests  a learner  frequently  and  provides  the  learner  with 
explicit  feedback  on  what  he  or  she  is  and  is  not  learning  supports  self  evaluation  and  compels  the  learner  to  review 
of  test  materials.  Computer  based  instruction  that  prompts  students  to  use  learning  processes  encourages  self 
regulation  (c.  f.,  Winne  & Stockley,  1998).  Instructors  who  initiate  frequent  graded  or  ungraded  tests  followed  by 
their  in-class  review  of  the  test  with  corrective  feedback  would  assure  learner  self  evaluation  and  test  review.  To 
further  compel  students  to  review  their  work,  instruction  might  require  learners  to  suggest  how  they  might  improve 
the  quality  of  a product  based  on  external  feedback  provided  through  the  instruction  or  the  instructor.  Another 
technique  especially  adaptable  to  distance  learners  would  be  to  use  the  same  performance  criteria  as  the  learner’s 
quality  control  checklist  during  assignment  preparation  and  as  an  evaluation  checklist  during  feedback  as  part  of  a 
multi-step  process:  (a)  provide  the  learner  with  a set  of  measurable  criteria  for  a product  or  process  in  the  format  of  a 
checklist,  (b)  instruct  the  learner  how  to  use  the  criteria  as  a quality  control  checklist  when  preparing  the  activity  or 
completing  the  product,  and  (c)  provide  the  learner  with  feedback  on  the  quality  of  the  learner’s  completed  process 
or  product  using  the  same  criteria  checklist  (Ley,  draft). 

Conclusion 

Several  distinct  activities  and  cognitive  processes  comprise  self  regulation  and  a select  few  of  them  differ 
between  higher  and  lower  achieving  adult  learners.  Low  achieving  learners  will  do  far  better  when  their  choices 
instructional  choices  are  limited  by  high  structure  (Kulik  & Kulik,  1991).  Lower  achieving  learners  are  more  likely 
to  be  poor  self  regulators  and  to  benefit  from  the  high  structure  inherent  to  the  SR  principles.  The  SR  principles 
guide  the  design  process  so  that  the  learner  is  not  left  to  choose  his  or  her  learning  environment.  Although  SR  is  to 
some  extent  context  dependent,  instructional  designers  should  consider  using  a framework  of  general  principles  that 
have  been  developed  to  address  the  distinct  SR  deficiencies  associated  with  achievement  levels  among  adult  learners. 
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Abstract 

Many  educational  institutions  face  a variety  of  pressures  to  adopt  emerging  technologies  in  their  every-day 
endeavors.  If  emerging  technologies  are  adopted  without  adequately  considering  the  systems  in  which 
they  are  to  be  used,  the  results  can  be  less  than  desired.  This  paper  describes  one  College's  efforts  to 
effectively  adopt  emerging  technologies  on  a general  basis.  The  various  systems  which  come  into  play  in 
this  context  are  discussed,  along  with  a theoretical  model  examining  faculty  needs  when  adopting 
emerging  technologies  in  education.  Educational  institutions  must  support  faculty  on  many  fronts  for 
successful  adoption  of  emerging  technologies. 


Educational  institutions  face  many  difficulties  in  dealing  with  the  adoption  of  new  technologies  in  their 
educational  endeavors.  In  an  era  in  which  political  pressures  include  having  the  same  proportion  of  computers  on 
desktops  in  the  U.S.  as  in  Japan,  or  providing  every  school  child  with  access  to  the  internet,  educators  are  constantly 
forced  to  adopt  new  technologies,  if  for  no  other  reason  than  to  “keep  up.”  In  the  rush  to  adopt  and  change,  it  is  easy 
to  lose  track  of  basic  considerations,  such  as  how  the  changes  will  affect  learning,  if  at  all,  or  how  new  technologies 
will  be  supported.  This  paper  will  examine  the  on-going  efforts  of  one  college  to  address  the  multi-faceted  problems 
of  adopting  new  technologies  in  an  educational  endeavor,  and  will  do  so  in  the  context  of  general  systems  theory, 
theories  of  innovation,  and  our  own  hierarchy  of  faculty  needs  when  adopting  technology  in  education. 


Defining  Systems 

Merriam-Webster  (1998)  defines  "system"  as  "a  regularly  interacting  or  interdependent  group  of  items 
forming  a unified  whole."  Romiszowski  (1981)  points  out  that  "a  system  exists  because  we  have  chosen  to  consider 
it  as  that.”  Using  a bicycle  as  an  example,  he  points  out  that  a bicycle  might  be  considered  part  of  any  number  of 
systems  (a  transportation  system,  for  example).  A bicycle  might  also  be  considered  a system  itself,  and  might  be  said 
to  contain  sub-systems  (such  as  the  derailleur  and  associated  shifting  mechanisms). 

Systems  interact.  In  general,  that  which  interacts  with  a system  might  be  called  an  "input"  and  the  result  of 
the  interaction  (what  any  given  system  produces  as  the  result  of  an  input)  is  commonly  called  an  "output."  For  a 
current  example,  a public  school  might  be  considered  an  educational  "system";  having  a new  internet  connection 
installed  in  every  room  might  be  considered  an  input  to  that  system.  Those  who  push  for  and  pay  for  the  new 
internet  connections  are  probably  hopeful  that  the  resulting  output  will  be  higher  grades,  or  higher  mean  scores  on 
standardized  tests,  or  "better  educated"  children.  In  the  context  of  this  paper,  the  university  functions  as  the  macro 
system,  of  which  various  micro  systems,  such  as  colleges  and  departments,  are  a part.  The  inputs  and  outputs  we  will 
examine  have  to  do  with  the  adoption  and  implementation  of  new  technologies. 

To  approach  a problem  "systemically"  entails  more  than  simply  acknowledging  that  the  context  for  the 
problem  in  question  can  be  considered  a system  which  interacts  with  other  systems.  A systemic  approach  also  entails 
examining  a system's  inputs  and  outputs  and  attempting  to  make  changes  that  result  in  (hopefully)  better,  or  more 
acceptable  outcomes.  If  such  an  examination  is  done  as  the  result  of  a problem,  it  frequently  follows  a pattern. 
Romiszowski  (1981),  provides  a fairly  representative  pattern  of  a systemic  problem-solving  process:  a)  defining  the 
problem,  b)  analyzing  the  problem,  c)  selecting  and  synthesizing  an  optimal  solution,  d)  implementing  the  solution  in 
a controlled  setting,  and  e)  evaluating  and  revising  the  solution.  This  pattern  illustrates  the  basic  structure  for  any 
number  of  systemic  instructional  design  models  frequently  employed  by  professionals  in  our  field  (Gustafson  & 
Powell,  1991). 


Identifying  the  Systems  in  Question 

A complete  description  of  our  instructional  support  context  has  been  provided  elsewhere  (Danielson  & 
Burton,  1999)  to  which  the  interested  reader  can  refer  if  desired.  The  description  here  will  be  limited  to  providing 
that  information  necessary  to  understand  the  systems  involved. 

The  Macro-System 

The  overall  system  to  which  we  will  refer  is  Virginia  Polytechnic  and  State  University  (Virginia  Tech),  a 
large  land  grant  university  in  south-west  Virginia.  The  administration  at  Virginia  Tech  has  long  valued  the 
importance  of  remaining  abreast  of  emerging  technologies.  In  keeping  with  these  values,  beginning  in  the  early 
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nineteen  nineties,  the  administration  at  Virginia  Tech  decided  that  it  would  be  advantageous  Jo  providefaculty  of  the 
- University  with  computers  to  aid  in  their  instructional  endeavors;  This  was  realized  through  a program  known  as  the 
faculty  Development  Institute  (FDI). 

The  goal  of  FDI  was  two-fold.  First,  FDI  would  place  a computer  on  each  faculty  member’s  desk  every  5 
years.  Second,  FDI  would  provide  faculty  with  the  knowledge  and  skills  necessary  to  use  their  computers  for 
developing  courseware.  In  exchange  for  receiving  a new  computer,  each  faculty  member  would  be  required  to  attend 
a series  of  3-5  all-day  workshops,  aimed  at  teaching  them  how  to  use  their  new  computers  effectively.  In  a related 
effort,  State  Council  of  Higher  Education  in  Virginia  (SCHEV)  monies  were  made  available  through  the  University 
to  departments  to  spend  on  hard  "technology"  as  they  saw  fit.  In  systems  terms,  the  hope  was  that  if  hardware, 
software,  and  a training  session  went  "in"  (to  the  college/department  systems)  effective  technology-assisted 
instruction  would  come  "out." 

The  Sub-Systems 

If  the  University  is  the  macro  system  in  the  context  of  this  paper,  the  sub-system  of  interest  directly 
subordinate  to  it  is  the  College  of  Human  Resources  and  Education  (CHRE)  (once  two  colleges  which  were  forcibly 
merged  shortly  after  the  advent  of  FDI).  Faculty  in  the  college  returned  from  FDI  with  brand-new  computers  and  a 
generally  superficial  knowledge  of  how  to  use  presentation,  e-mail,  and  web-browsing  software.  The  technology  was 
"in"  the  system  and  it  was  time  to  wait  and  see  what  came  out. 

Sub-systems  (departments  and  programs)  within  the  college  played  an  important  role.  Some  departments 
(including,  generally,  several  from  the  education  "side")  used  their  SCHEV  money  to  maintain  and  build  on  what 
was  initiated  with  FDI.  Computers  were  replaced  when  outdated,  and  efforts  were  made  to  keep  software  up  to  date. 
Other  departments,  on  the  other  hand,  tended  to  allocate  their  SCHEV  moneys  in  other  ways,  leaving  faculty  and 
staff  with  old  systems  (both  in  terms  of  hardware  and  software)  which  became  unreliable  and  difficult  to  maintain 
and  support.  Many  faculty  rarely  or  never  used  their  computers  for  every-day  tasks,  much  less  to  create  and  maintain 
courseware.  Training  that  was  not  immediately  useful  was  quickly  forgotten.  Even  when  the  training  might  have 
been  considered  useful  by  some  faculty,  it  was  frequently  not  easily  implemented.  For  example,  the  first  several 
groups  of  faculty  receiving  FDI  training  learned  how  tty  do  e-mail  at  a time  when  there  were  very  few  others  to 
correspond  with  via  e-mail. 

Those  faculty  who  made  an  effort  to  use  their  new  equipment  and  skills  soon  discovered  other  problems. 
While  FDI  had  provided  them  with  computers  and  training,  it  did  not  provide  for  follow  up  support  to  deal  with  the 
myriad  of  technical  problems  associated  with  computer  use.  Some  professors,  frustrated  by  incessant  system 
crashes,  inability  to  network  correctly,  inability  to  print  reliably,  etc.,  abandoned  the  thought  of  using  their  computers 
for  course  development.  Some  professors  abandoned  their  computers  altogether,  and  found  vacancies  for  them  on 
the  floors  of  their  offices. 

Figure  1,  below,  illustrates  what  had  occurred.  The  University  "system"  had  introduced  a new  input  into 
several  sub-systems,  with  the  hope  that  effective  technology  based  instruction  would  emerge.  As  will  probably 
surprise  few  experts  in  IT,  however,  the  problem  was  not  so  simple,  and  the  results  did  not  meet  hoped  for 
expectations. 

The  fact  that  FDI  alone  did  not  result  in  faculty  using  emerging  technologies  effectively  for  teaching  can 
probably  be  explained  relatively  easily.  Scholars  and  practitioners  in  IT  generally  understand  and  accept  that  hard 
technology  is  not  an  educational  panacea.  Most  IT  experts  could  glance  at  figure  1 below  and  immediately  suggest 
other  "inputs"  that,  combined  with  hard  technology  and  limited  training  would  be  more  likely  to  result  in  effective 
technology-assisted  training  outcomes.  Most  would  even  suggest  that,  in  the  absence  of  an  effective  front-end 
analysis  based  on  a well-defined  "problem"  any  particular  set  of  inputs  amount  to  a "shot  in  the  dark"  at  an 
unidentified  target. 

In  this  context,  however,  those  working  in  the  sub-systems  of  Virginia  Tech  were  not  generally  given  the 
option  of  determining  the  "inputs"  (from  the  macro  system)  but  were  rather  left  to  use,  (or  not  use)  what  they  got. 
Recognizing  the  difficulty  faculty  were  facing  in  attempting  to  put  FDI-gained  knowledge  and  tools  to  work, 
administrators  in  the  College  of  Human  Resources  and  Education  determined  to  address  this  multi-faceted  problem 
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Figure  1.  From  the  macro  system  into  the  sub-system 


The  most  immediate  problem  facing  the  college  appeared  to  be  of  a technical  nature.  Faculty  were  not 
using  their  newly  acquired  technology  because  they  couldn’t  get  it  to  work,  or  didn't  know  how  to  use  it.  Frequently, 
they  turned  to  office  staff  for  technical  help,  or  to  act  as  a buffer  between  themselves  and  their  technology.  (One 
faculty  member,  for  example,  would  print  electronic  mail  messages  and  ask  a secretary  to  re-type  them  into  a word 
document,  because  the  faculty  member  was  unfamiliar  with  the  cut  and  paste  function.)  Meanwhile,  office  staff 
struggled  with  technical  problems  of  their  own.  Frequently,  they  were  unable  to  keep  their  own  machines 
functioning  properly,  much  less  maintain  machines  for  faculty  members  while  still  performing  their  regular  work. 

A Compensating  Sub-System  Introduced  — "Housecalls" 

In  an  attempt  to  resolve  the  myriad  of  technical  problems  plaguing  the  college,  administrators  at  the  College 
level  and  within  the  department  of  Teaching  and  Learning  determined  to  create  a service  which  would,  a)  provide 
technical  support  for  the  college,  b)  provide  a venue  for  training  and  supporting  graduate  students,  and  c)  support 
faculty  in  their  efforts  to  use  their  computers  as  educational  tools.  With  this  decision,  the  ’’housecalls”  program  was 
created. 

The  initial  and  primary  focus  of  the  housecalls  program  was  to  provide  reliable,  prompt  (contact  within  the 
hour),  and  personal  technical  assistance  to  faculty  within  the  college.  Housecalls  was  not  to  be  merely  a technical 
service,  however.  It  was  to  provide  “human  contact”,  (97%  of  housecalls  calls  are  dealt  with  face  to  face),  advice 
and  on-the-spot  training  for  technical  and  instructional  questions,  and  in-service  training  for  technical  and 
instructional  issues.  The  following  points  summarize  the  housecalls  service  as  it  exists  in  the  CHRE  (8/96  - 11/98). 

1 . Housecalls  serves  the  College  of  Human  Resources  and  Education  including  approximately  1 70  faculty  and  65  staff  using 
about  400  machines  (Mac  and  IBM  compatible). 

2.  Housecalls  employs  six  GA’s  (at  approximately  $15,000.00/yr.  = $90,000.00),  one  staff  member  (approximately 
$21,000.00/yr.),  and  other  wage  employees  (approximately  $10,000.00/yr.).  Operating  and  inventory  costs  are 
approximately  $15, 000. 00/yr.  The  total  costs  reach  approximately  1 35,000. 00/yr. 

3.  Housecalls  handles  on  average  124  calls  per  month.  Less  than  3%  are  not  handled  in  person,  and  less  than  1 % are  referred 
to  another  level  of  maintenance.  Time  per  call  ranges  from  2 minutes  to  7+  hours.  Technical  difficulty  of  calls  ranges 
from  wiggling  a wire  to  reformatting  a hard  drive  and  reinstalling  all  software.  Level  of  difficulty  of  training-type  calls 
ranges  from  teaching  someone  how  to  increase  margins,  to  helping  someone  design  an  effective  course  web  page  or  design 
an  effective  course  strategy  for  asynchronous  communication.  Level  of  difficulty  of  calls  increases  as  clients  learn  and 
become  more  sophisticated. 
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Returning  then,  to  our  systems  analogy,  figure  2 illustrates  the  rok  of  housecalls  — introduced  as  a 
compensating  system  at  the  College  level  to  provide  additional  inputs  to  those  introduced  through  FDI.  The 
administrators  who  introduced  the  housecalls  program  saw  it  as  a too!  to  not  only  solve  technical  problems,  but  to 
also  move  the  college  in  the  direction  of  addressing  the  goal  that  led  to  FDI  that  of  seeing  faculty  use  emerging 
technologies  effectively  in  their  teaching.  This  strategy  has  been  initially  successful,  as  will  be  seen. 


Our  Context  and  Theory  of  Diffusion  and  Adoption  of  Educational  Technology 

Before  examining  the  specific  results  of  the  housecalls  program,  it  seems  appropriate  to  examine  it  from  the 
perspective  of  research  and  theory  in  the  area  of  diffusion  and  adoption  of  emerging  technologies  in  education.  In 
his  summary  of  such  research,  Holloway  (1996)  concludes  that  three  factors:  training,  resources,  and  involvement, 
are  consistently  shown  to  be  instrumental  in  the  adoption  of  emerging  technologies  by  educators.  The  housecalls 
program's  focus  is  primarily  on  providing  the  former  two  (in  the  form  of  technical  support),  and  not  on  the  third 
(encouraging  faculty  involvement  in  adopting  emerging  technologies).  Such  an  emphasis  seems  appropriate  in  our 
context  for  two  reasons.  First,  it  hardly  seems  appropriate  for  educational  technologists  to  "push"  hard  technology 
use  for  its  own  sake  in  any  situation.  Second,  research  seems  to  indicate  that  teachers  and  faculty  are  most  moved  to 
adopt  emerging  technologies  in  their  educational  endeavors  when  they  see  peers  doing  so  (as  opposed  to  being 
"pushed"  to  do  so  from  the  outside)  (Holloway,  1996). 

Rogers,  (1971)  one  of  the  pioneers  in  the  field  of  diffusion  research,  introduced  five  attributes  of 
innovations:  relative  advantage,  compatibility,  complexity,  trialability,  and  observability.  Ely,  (1990)  building  on 
the  work  of  Rogers  and  others,  identified  eight  conditions  that  facilitate  the  implementation  of  Educational 
Technology.  His  work  is  still  consistent  with  more  recent  findings,  and  his  eight  conditions  provide  a convenient 
framework  for  analyzing  our  context.  We  will  briefly  examine  each  of  Ely's  conditions,  providing  examples  from 
our  context  which  demonstrate  the  presence  of  these  conditions.  Those  conditions  are:  a)  dissatisfaction  with  the 
status  quo,  b)  knowledge  and  skill  exist,  c)  resources  are  available,  d)  time  is  available,  e)  rewards  or  incentives  exist 
for  participation,  f)  participation  is  expected  and  encouraged,  g)  commitment  by  those  who  are  involved,  h) 
leadership  is  evident. 

Housecalls  and  Ely's  Eight  Conditions 

1.  Dissatisfaction  with  the  Status  Quo:  Clearly,  most  faculty  and  staff  in  the  CHRE  were  dissatisfied 
with  a situation  in  which  they  were  expected  to  use  computers  without  technical  support.  (It  is  unclear 
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the  extent  to  which  initial  dissatisfaction  led  faculty  members  to  take  FDI  machines  in  the  first  place). 
The  housecalls  service  does  address  faculty  dissatisfaction  with  technical  problems.  Increasingly, 
dissatisfaction  in  other  areas  is  leading  to  other  kinds  of  implementation  as  well.  For  example,  at 
Virginia  Tech  more  and  more  students  are  demanding  the  luxury  of  having  notes  and  outlines,  etc.  on- 
line. This  is  causing  some  faculty  to  change  in  ways  that  they  probably  would  not  have  otherwise. 

2.  Knowledge  and  Skill  Exist:  The  housecalls  program  is  staffed  by  graduate  students  in  the 

Instructional  Technology  program  in  the  Department  of  Teaching  and  Learning.  The  collective 
knowledge  and  skills  of  this  group  makes  providing  technical  assistance  possible. 

3.  Resources  are  Available:  Without  the  availability  of  money  from  the  State  Council  of  Higher 

Education  in  Virginia  (SCHEV)  equipment  trust  fund,  and  original  FDI-provided  computers,  most 
faculty  and  staff  would  not  be  able  to  support  original  purchase  of  or  continued  upgrading  of  equipment 
and  software.  In  the  department  of  Teaching  and  Learning,  all  faculty  and  staff  are  provided  with  a 
computer  to  take  home  and  work/play  on.  The  department  also  provides  a modem  and  pays  the 
monthly  internet  access  fee  for  faculty  and  staff. 

4.  Time  is  Available:  It  is  not  clear  that  faculty  at  Virginia  Tech  have  any  more  time  to  spend  adopting 
new  technologies  than  faculty  anywhere  else.  Indirectly,  however,  the  housecalls  program  does  "give” 
faculty  time,  because  those  who  use  hard  technology  can  spend  less  time  fighting  trivial  technical 
problems. 

5.  Rewards  or  Incentives  Exist  for  Participation:  Faculty  members  receive  new  computers  for 
participating  in  FDI.  Increasingly,  there  is  anecdotal  evidence  to  suggest  that  faculty  are  rewarded  for 
adopting  emerging  technologies  and  effective  instructional  design  strategies  in  the  form  of  higher 
approval  from  students.  There  are  formal  incentive  structures  which  encourage  adopting  emerging 
technologies  in  instruction  as  well.  The  university  supports  a grant  program  which  awards  monies  for 
an  “innovations  in  learning  with  technology”  program.  Similarly,  the  College  of  Human  Resources  and 
Education  has  an  Associate  Dean  of  Innovations,  who  allocates  monies  for  the  implementation  of 
instructional  technology.  It  is  not  clear,  however,  whether  or  not  faculty  efforts  in  this  area  will  be 
rewarded  in  the  area  of  promotion  and  tenure. 

6.  Participation  is  Expected  and  Encouraged:  Faculty  members  are  required  to  participate  in  FDI. 
Various  innovations  (such  as  the  use  of  e-mail)  are  also  required  on  a department  by  department  basis. 

7.  Commitment  by  those  who  are  involved:  Because  the  housecalls  program  was  started  at  the  College 
level,  and  because  FDI  was  started  at  the  University  level,  there  is  commitment  to  the  success  of  each 
from  administrators.  Similarly,  in  the  case  of  the  housecalls  program,  because  the  services  provided 
are  so  universally  required,  there  is  general  commitment  to  its  continued  success.  The  fact  that  the 
housecalls  program  continues  to  be  funded,  in  spite  of  the  non-trivial  costs  involved,  is  perhaps  the 
surest  indication  of  commitment  on  the  part  of  decision  makers. 

8.  Leadership  is  Evident:  Various  leaders  have  proven  instrumental  in  the  support  of  the  housecalls 
program,  and  in  the  adoption  of  emerging  technologies  in  the  College  of  Human  Resources  and 
Education.  At  the  University  level,  the  provost  has  stressed  the  importance  of  utilizing  new 
technologies  as  much  as  possible.  At  the  College  level,  the  Dean  of  Innovations  has  consistently 
supported  and  funded  initiatives  to  integrate  technology  in  teaching  and  learning  endeavors.  The 
department  chair  of  the  department  of  Teaching  and  Learning  has  also  been  instrumental.  He 
introduced  and  championed  the  housecalls  program,  and  has  consistently  supported  the  department  of 
teaching  and  learning  in  adopting  new  technologies.  He  has  also  stressed  the  importance  of  thoughtful 
application  of  technology,  acknowledging  the  weaknesses  of  using  technology  for  technology’s  sake, 
and  stressing  the  importance  of  supporting  faculty  regardless  of  their  support  needs. 

Hierarchy  of  Faculty  Needs  when  Adopting  Technology  in  Education 

Our  experience  with  the  housecalls  program  and  the  intriguing  set  of  circumstances  leading  to  its  creation 
have  led  us  to  conceptualize  a model  for  conveying  faculty  needs  in  adopting  technology  in  higher  education.  If 
research  suggests  that  training,  resources,  and  involvement  are  required  for  the  adoption  of  emerging  technologies  in 
education,  then  our  model  assumes  involvement  (i.e.,  faculty  are  seeking  to  use  emerging  technologies  already,  — we 
are  not  pushing  it)  and  suggests  an  order  in  which  training  and  resources  might  most  effectively  be  provided. 

In  early  1999  we  introduced  a “hierarchy  of  instructional  technology  needs”  (Danielson  & Burton,  1999) 
reminiscent  of  Maslow’s  (1970)  hierarchy  of  needs.  Maslow’s  hierarchy  illustrates  the  theory  that  meeting  certain 
fundamental  human  needs  is  prerequisite  to  meeting  others.  In  his  hierarchy,  the  need  for  food  and  shelter  is 
fundamental,  and  is  prerequisite  to  meeting  higher  needs  such  as  the  need  for  human  affection. 

In  our  analogy  to  the  implementation  of  educational  technology,  certain  basic  technological  needs  precede 
and  are  prerequisite  to  others.  Our  original  version  of  the  hierarchy  of  faculty  needs  when  adopting  technology  in 
education  appears  in  figure  3 below.  As  can  be  seen,  we  proposed  that  the  most  basic  of  all  requirements  (if  faculty 
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are  to  use  these  tools  effectively)  is  that  of  having  equipment  functioning  properly.  ^The  highest  level  requirement  is 
that  of  creating  good,  theory-based  instructional  products.  Clearly,  while  the  latter  is  the  ultimate,  and  certainly  more 
glamorous  goal,  we  argue  that  it  will  not  happen  until  the  requirements  below  it  have  been  met.  It  has  also  been  our 
experience  that  the  needs  which  require  the  most  time  to  address  are  those  found  at  the  top  and  the  bottom  of  the 
pyramid.  Therefore,  any  endeavor  which  strives  to  meet  these  requirements  must  have  the  ability  to  spend  a great 
deal  of  time  and  resources  “at  the  bottom”  before  moving  its  way  up.  Furthermore,  we  argued  that  it  is  futile  to  “sit 
at  the  top”  of  the  pyramid,  so  to  speak,  and  wait  for  faculty  to  come  ask  for  assistance  with  weighty  and  important 
matters  of  theory  and  development.  This  is  the  case  because,  a)  most  faculty,  without  support  at  the  bottom,  will 
never  arrive  at  the  top,  and  b)  those  faculty  who  have  had  the  drive  to  arrive  to  the  top  unaided  are  unlikely  to  solicit 
help  once  there. 


* Labor  intensive  at  top  and  bottom 


Supporting  and  Enhancing  the  Hierarchy 

Since  conceiving  of  the  hierarchy  presented  above,  we  have  had  an  additional  year  to  examine  our 
assumptions,  collect  additional  relevant  data,  watch  the  housecalls  program  at  work,  and  talk  with  faculty.  We  have 
observed  a fair  amount  of  anecdotal  evidence  supporting  the  underlying  assumptions  of  the  hierarchy.  Our 
experience  with  the  housecalls  program,  as  well  as  our  conversations  with  faculty  members  have  also  led  us  to  make 
several  adjustments  to  the  bottom  of  the  hierarchy.  First,  because  there  is  clearly  an  important  distinction  between 
the  presence  of  hard  technology  (e.g.  hardware  and  software),  and  "working  technology"  (which  implies  that  the 
technology  gets  fixed  when  it  breaks),  we  have  chosen  to  divide  the  "working  technology"  section  into  two  sections: 
a)  presence  of  hard  technology,  and  b)  working  technology.  Second,  because  the  necessity  of  time  is  such  a 
prevalent  part  of  relevant  literature,  and  because  we’ve  found  it  mentioned  so  frequently  by  faculty  as  an  essential 
factor,  we  have  also  chosen  to  include  it  as  a "need"  on  the  hierarchy.  The  adjusted  hierarchy  appears  in  figure  4, 
below. 

Data  supporting  our  assumptions  can  best  be  summarized  through  general  statistics  about  the  housecalls 
service,  highlights  from  several  in-depth  interviews  with  faculty  from  the  CHRE,  and  finally,  several  general  reports 
of  how  similar  programs  are  being  implemented  elsewhere. 
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Figure  4.  New  Hierarchy  of  Faculty  Needs  When  Adopting  Technology  in  Education 


A 

Theory-based  instructional 
software  use  and  development 


General  Statistics  on  the  Ho  use  calls  Service 

Observing  the  trend  of  housecall  use  since  its  introduction  in  August  of  1996  reveals  the  following: 

1.  Since  its  introduction  in  1996,  the  requirement  for  housecalls  services  has  remained  stable. 
Housecalls  handles,  on  the  average,  124  housecalls  per  month.  However,  housecalls  staff  indicate 
that  the  nature  of  the  calls  is  becoming  more  complex.  This  could  indicate  that,  while  faculty  and 
staff  are  becoming  more  sophisticated  and  calling  on  housecalls  less  for  trivial  service,  they  are 
also  managing  to  "get  into  more  trouble”  (i.e.,  their  use  is  more  sophisticated,  and,  therefore,  their 
problems  have  become  more  sophisticated  as  well). 

2.  While  the  need  for  basic  technical  support  has  remained  solid,  there  has  been  a gradual  increase  in 
requests  for  services  "higher  up  the  hierarchy.”  Lately,  this  has  come  mostly  in  the  form  of  faculty 
requests  for  help  designing  and  "putting  up”  web  pages.  We  believe  that  this  indicates  a couple  of 
things:  a),  faculty  and  staff  are  learning  how  to  solve  some  of  their  simpler  low-tech  problems, 
making  them  more  willing  to  take  on  "higher-tech”  ones,  b)  Faculty  and  staff  have  developed  a 
trust  for  housecalls  and  are  willing  to  call  on  them  for  one-to-one  training,  c)  Faculty  and  staff  are 
actually  doing  more  sophisticated  instruction  with  their  technology  (i.e.,  they’re  higher  up  the 
pyramid)  than  they  used  to  be.  This  is  due,  in  part  at  least,  to  the  fact  that  they  aren’t  being  held 
back  by  lower-level  problems.  Interviews  with  faculty  indicate  that  web  page  development  is  also 
partly  due  to  pressure  felt  from  students  who  are  demanding  more  availability  of  services  on-line. 

Interviews  with  Faculty 

Four  faculty  members  who  have  called  upon  housecalls  for  help  with  higher-level  tasks  were  interviewed  at 
length  regarding  their  technology  use  in  their  teaching  endeavors.  These  faculty  come  from  a variety  of 
backgrounds.  Two  have  taught  several  televised  courses.  Another  has  taught  a course  that  was  conducted  strictly 
on-line.  The  other  is  the  editor  and  manager  of  an  on-line  journal.  All  also  use  (to  varying  degrees)  electronic 
presentation  software  and  web  pages  in  their  instruction. 

Several  basic  trends  could  be  found  throughout  all  the  comments  of  the  faculty  members.  First,  all  agreed 
that  basic  support  in  the  form  of  university-supplied  computers  and  technical  assistance  is  very  important,  (and  in 
three  of  the  four  cases)  essential,  to  their  adoption  of  emerging  technologies  in  education.  None  of  the  faculty 
members  expressed  willingness,  with  their  own  money,  to  provide  themselves  (at  work)  with  a computer  equivalent 
to  the  one  that  had  been  supplied  by  the  university. 

Second,  all  faculty  members  (consistent  with  Ely  and  others)  indicated  that  the  availability  of  paid  time  to 
develop  technology-assisted  courses  is  a great  help  in  doing  so,  and  that  the  lack  of  time  is  a major  deterrent.  One 
professor  commented  that  he  was  able  to  build  an  on-line  course  because  the  University  bought  out  his  classes  for  a 
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semester  so  that  he  could  have  time  to  dedicate  to  the  course  he  was  building.  Two  mentioned  that  they  have  not 
been  able  to  make  desired  technological  enhancements  to  courses  they  teach  for  lack  of  time  to  do  so. 

Similarly,  all  faculty  stressed  the  importance  of  the  housecalls  program.  All  four  mentioned  both 
housecalls'  training  and  technical  support  functions.  All  mentioned  the  convenience  of  having  a housecalls 
representative  available  the  same  day  (and  frequently  the  same  hour)  as  their  request  for  assistance  to  deal  with 
problems.  Issues  that  they  felt  they  would  have  spent  hours  resolving  on  their  own,  they  were  able  to  refer  to 
housecalls,  and  spend  their  time  on  other  endeavors.  Three  faculty  members  went  so  far  as  to  say  that  they  would 
not  have  been  able  to  use  the  technology  they  are  using  without  the  support  of  housecalls  (or  a similar  service),  and 
the  other  acknowledged  that  attempting  to  do  what  they  had  done  without  housecalls  would  have  been  very  difficult. 

The  faculty  members  interviewed  also  reported  relying  heavily  on  housecalls  for  learning  how  to  use  their 
teaching  tools.  For  example,  one  learned  how  to  design,  build  and  "put-up"  a homepage  through  several  hour-long 
sessions  with  a housecalls  staff  member.  Another  spent  similar  long  hours  getting  questions  answered  relating  to 
how  to  use  the  Mac  operating  system  to  transfer  files,  and  then,  after  switching  to  a PC  platform,  how  to  use  the  PC 
for  file  transfer  — all  within  the  context  of  building  a web  page  for  a class. 

Finally,  housecalls  was  considered  important  for  its  continuity.  After  a call  has  been  completed,  a 
housecalls  staff  member  calls  the  faculty  or  staff  member  who  was  served  to  verify  that  the  problem  was  solved  and 
check  for  subsequent  problems.  Records  are  kept  on  all  calls  made,  so  specific  recurring  problems  can  be  dealt  with 
in  a systematic  way.  Faculty  consider  this  kind  of  continuity  valuable. 

GeneralizabUity  of  Housecalls  and  the  Hierarchy 

Obviously,  our  observations  lack  statistical  generalizability.  However,  there  is  some  evidence  that  the 
principles  discovered  here  are  applicable  elsewhere.  Several  colleges  in  the  Appalachian  College  Association  have 
built  support  structures  for  their  colleges  patterned  after  the  housecalls  service.  The  following  observations  come 
from  administrators  at  two  of  those  colleges,  who  have  been  involved  in  such  programs. 

1-  The  two  programs  investigated  are  quite  similar  to  each  other.  Both  employ  10  staff  members,  and  serve  about  200 
hundred  faculty  and  staff  and  1000-1500  students.  Both  provide  software  and  hardware  support. 

2.  Both  programs  report  that  technology  use  by  those  supported  (ranging  from  e-mail  correspondence  to  having  and 
maintaining  web  pages)  has  increased  dramatically  since  the  services  began. 

3.  Both  programs  report  requests  from  faculty  and  staff  for  services  beyond  the  services  provided.  In  one  case,  faculty  are 
requesting  support  for  web  page  development  (which  is  not  provided).  In  the  other,  users  want  more  computers  available 
for  checkout,  and  service  of  personally-owned  computers  (which  is  not  provided). 

4.  Both  programs  report  that  faculty  are  using  the  new  support  services  to  employ  emerging  technologies  in  their 
instructional  efforts.  This  use  ranges  from  setting  up  web-sites  for  students  to  access,  to  using  presentation  software  for 
teaching  classes. 

In  general,  what  administrators  at  these  colleges  have  found  seems  consistent  with  the  idea  that  educators 
will  not  and  cannot  use  emerging  technologies  effectively  without  support,  and  that,  given  support,  they  will  be  more 
likely  to  use  emerging  technologies  in  their  teaching. 

Systems  and  the  Hierarchy  of  Faculty  Needs  when  Adopting  Technology  in  Education 
FDI  (The  Macro-System's  Contribution) 

We  will  return  now  to  our  context  at  Virginia  Tech,  and  the  systems  that  we  initially  examined.  We 
suggested  that  the  university  might  be  considered  a macro-system  of  which  Colleges  (in  our  case  the  CHRE), 
departments,  and  programs  play  a role  as  sub-systems.  In  summarizing  the  University’s  initiative  to  encourage 
faculty  use  of  technology  in  teaching,  the  university  made  available  money  for  computer  equipment,  and  a training 
program  in  the  form  of  3-day  to  one-week  seminars  (FDI).  Comparing  this  to  our  hierarchy  (see  figure  5 below)  this 
seems  an  appropriate  first  step  in  promoting  the  use  of  emerging  technologies  in  education.  This  is  the  case  because 
the  most  basic  element  of  working  technology  (looking  at  the  bottom  of  the  pyramid)  is  hardware  and  software.  The 
two  upper  arrows  in  figure  5 illustrate  FDI's  initial  training  efforts,  and  the  bottom  arrow  demonstrates  the  initial 
hardware  and  software  in.  The  fact  that  the  FDI  program  was  inadequate  for  achieving  the  desired  outcome  (theory 
based  instructional  software  use  and  development),  does  not  negate  the  value  of  either  of  these  initial  inputs.  They 
simply  were  not  enough. 
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Figure  5:  FDl's  contribution  to  the  University’s  End  Goal 


Housecalls  ( One  Subsystem ’s  Contribution) 

The  fact  that  the  original  EDI  efforts  were  not  producing  the  desired  outcomes  led  administrators  in  the 
CHRE  to  introduce  new  system  "inputs."  The  input  in  this  case  was  a new  system,  "housecalls."  Figure  6,  below, 
illustrates  how  the  housecalls  system  specifically  contributed  to  meeting  specific  faculty  needs.  As  the  arrows 
reflect,  the  housecalls  program  has  provided  some  service  for  faculty,  at  nearly  every  level  of  the  hierarchy,  though, 
as  the  thickness  of  the  arrows  is  designed  to  indicate,  the  bulk  of  that  support  continues  to  focus  on  maintaining 
working  technology.  It  seems  clear,  nonetheless,  that  the  housecalls  service  has  helped  compensate  for  some  of  the 
deficits  left  by  the  FDI  program  in  moving  towards  the  ultimate  goal  of  theory-based  instructional  software  use  and 
development.  It  also  seems  clear  that  if  university  and  college  administrators  are  serious  about  wide-spread  theory- 
based  instructional  software  use  and  development,  there  are  still  some  pieces  of  the  puzzle  missing.  The  most 
notable  of  these  pieces  is  time.  While  the  housecalls  service  continues  to  "free-up"  faculty  time  somewhat,  it  still 
seems  clear  that  for  most  faculty  to  consistently  use  new  technologies  effectively,  they  must  be  given  time  to  do  so. 
We  also  anticipate  that  as  more  faculty  attempt  to  integrate  emerging  technologies  in  their  courses,  the  housecalls 
service  will  receive  an  increasing  number  of  requests  for  service  higher  up  the  hierarchy.  As  the  need  for  technical 
support  will  likely  remain  strong,  housecalls  will  likely  need  to  expand  somewhat  to  meet  the  increased  need,  or 
another  system  will  have  to  be  introduced  to  provide  more  specialized  service. 
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Figure  6:  Housecalls  and  FDI  Contribution  to  the  University's  End  Goal 


Contributions  by  FDI 


Conclusion 

Given  current  political  and  educational  climates,  there  may  be  any  number  of  reasons  why  educators  might 
be  expected  to  adopt  emerging  technologies  in  their  educational  endeavors.  Our  hope  is  that  regardless  of  the 
context,  the  hierarchy  we  have  provided  might  prove  useful  to  practitioners  who  have  "pieces”  of  a technology 
solution,  or  who  are  trying  to  put  one  together.  We  contend  that  anyone  seeking  to  adapt  emerging  technologies  in 
an  educational  endeavor  must  take  into  consideration  all  the  "technology"  needs  faculty  will  face,  and  the  order  in 
which  those  needs  are  addressed. 

References 

Danielson,  J.  A.,  & Burton,  J.  K.  (1999).  A support  system  for  instructional  technology  in  higher 
education.  In  R.  M.  Branch  & M.  A.  Fitzgerald  (Eds.),  Educational  media  and  technology  yearbook  (Vol.  24, ). 
Englewood,  Colorado:  Libraries  Unlimited. 

Ely,  D.  P.  (1990).  Conditions  that  facilitate  the  implementation  of  educational  technology  innovations. 
Journal  of  Research  on  Computing  in  Education.  23(2),  298-305. 

Gustafson,  K.  L.,  & Powell,  G.  C.  (1991).  Survey  of  instructional  development  models  with  an  annotated 
ERIC  bibliography  (Information  Analyses-  ERIC  Clearinghouse  Products  (071)  - Reference  Materials  -- 
Bibliographies  (131)  ED  335  027).  Syracuse,  NY:  Syracuse  University. 

Holloway,  R.  E.  (1996).  Diffusion  and  adoption  of  educational  technology:  A critique  of  research  design. 

In  D.  H.  Jonassen  (Ed.),  Handbook  of  research  for  educational  communications  and  technology  (pp.  1107-1133). 
New  York:  Simon  & Schuster  Macmillan. 

Maslow,  A.  H.  (1970).  Motivation  and  Personality.  ( (2nd  ed.)  ed.).  New  York:  Harper  and  Row. 

Merriam-Webster.  (1998).  WWWebster  dictionary  f on-line] . Meri am- Webster.  Available:  http://www.m- 
w.com/mw/netdict.htm  [1998,  April  1998]. 

Rogers,  E.  M.  (1971).  Communication  of  innovations.  New  York:  The  Free  Press. 

Romiszowski,  A.  J.  (1981).  Designing  instructional  systems.  London:  Kogan  Page. 


110 


Navigational  Structure  on  the  World  Wide  Web:  Usability  Concerns,  User 
Preferences,  and  “Browsing  Behavior” 

Theodore  Frick 
John  A.  Monson 
Richard  F.  Xaver 
Gulsen  Kilic 
Aaron  T.  Conley 
and 

Beatrice  Wamey 

School  of  Education,  Indiana  University 


Abstract 

A primary  consideration  in  web  site  design  is  that  the  site  be  easily  navigable  for  its  intended  audience. 
For  " informational " sites,  where  the  primary  purpose  is  to  provide  information  about  a service  or 
program , a menu  structure  is  often  used  to  direct  the  users  to  different  areas  of  the  site.  There  are  several 
approaches  a site  designer  considers  in  developing  a menu  structure.  One  consideration  is  the  content  of 
the  menus  themselves  (what  choices  are  available  to  the  user).  Another  consideration  is  the  physical 
layout  of  the  menu  structure  itself  The  physical  layout  of  a menu  may  be  described  as  being  one  of  at  least 
three  different  types.  The  "traditional,"  hierarchical  menu  structure  is  used  most  often.  Persistent  menus,  or 
menus  which  remain  on  the  screen,  are  often  accomplished  by  the  use  of  HTML  tables  or  frames.  A third 
approach  is  the  use  of  graphic  or  textual  "maps"  which  show  the  user  the  path  taken  to  a particular 
document.  The  purpose  of  this  study  was  to  determine  whether  visitors  who  are  "just  browsing"  can  do  so 
more  effectively  under  one  of  the  three  menu  schemas  described  above  (hierarchical  disappearing  menus, 
persistent  menus  via  frames  or  tables,  and  representations  of  navigational  maps). 

Introduction 

A primary  consideration  in  web  site  design  is  that  the  site  be  easily  navigable  for  its  intended  audience.  For 
"informational"  sites,  where  the  primary  purpose  is  to  provide  information  about  a service  or  program,  a menu 
structure  is  often  used  to  direct  the  users  to  different  areas  of  the  site. 

There  are  several  approaches  a site  designer  considers  in  developing  a menu  structure.  One  consideration  is 
the  content  of  the  menus  themselves  (what  choices  are  available  to  the  user).  This  is  often  accomplished  through  a 
combination  of  intuition  and  investigation.  If  one  of  the  purposes  of  a site  is  to  provide  information  that  would 
normally  be  available  through  other  means,  such  as  direct  inquiry  to  the  information  provider  via  a telephone  call, 
consultation  with  those  responsible  for  such  inquiries  can  provide  a well-established  listing  of  frequently  asked 
questions  and  answers.  Another  consideration  is  the  physical  layout  of  the  menu  structure  itself.  This  consideration 
is  often  less  dependent  on  inquiry  or  even  intuition;  it  may  be  that  aesthetics,  visual  design,  and  the  limitations  of 
servers  and  browsers  determine  how  a menu  is  presented.  The  physical  layout  of  a menu  may  be  described  as  being 
one  of  at  least  three  different  types: 

1.  The  "traditional,"  hierarchical  menu  structure  is  used  most  often.  In  this  structure,  menus  are  "pick  lists"  which  move  a 
user  through  a number  of  choices.  At  each  step,  the  content  of  the  previous  menu  disappears  and  is  replaced  by  its 
subsequent  menu. 

2.  Persistent  menus,  or  menus  which  remain  on  the  screen,  are  often  accomplished  by  the  use  of  HTML  tables  or  frames. 
Frames  are  actually  sets  of  HTML  pages  which  appear  together.  An  advantage  of  this  approach  is  that  the  menu  can 
remain  on  the  screen  at  all  times,  regardless  of  the  content  of  the  subsequent  pages.  In  other  words,  the  menu  remains 
within  one  frame,  and  choices  made  via  that  menu  are  displayed  within  another  frame.  The  disadvantage  of  this  approach 
is  that  not  all  web  browser  clients  can  support  frames;  some  usability  experts  actively  discourage  the  use  of  frames  because 
they  tend  to  change  the  metaphor  of  the  web  page  and  can  be  confusing  to  some  users.  (Nielsen  J.,  1996).  In  an  article  by 
Andrews  (1996)  web  designer,  Frank  Forbes,  offered  both  frame  and  non-frame  versions,  his  finding  was  that  only  1 of 
the  10  users  preferred  the  frames  version 

3.  A third  approach  is  the  use  of  graphic  or  textual  "maps"  which  show  the  user  the  path  taken  to  a particular  document.  An 
example  of  that  type  of  design  is:  http://www.useit.com/alertbox/96l2.html.  On  that  design  type,  the  path  one  followed  is 
shown  as  an  active  link  at  the  top  of  each  page.  This  may  help  the  user  to  see  where  he  is,  how  he  reached  there,  and 
enable  him  to  return  to  directly  to  the  level  he  wants. 


In  determining  which  approach  is  most  useful  and  usable  for  visitors  to  a web  site,  designers  often  employ 
development  and  test  procedure  which  can  spot  strengths  and  weaknesses  of  an  emerging  system.  One  process  for 
doing  this  is  described  in  Practical  Web  Development:  A Systematic  Process  (Boling  & Frick,  1998). 
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Needs  analysis  determines  the  typical  questions  users  desire  answers  for  and  guides  the  development  of  the 
structure,  layout,  andcontent.  Paper  prototypes  are  developed  for  the  proposed  site.  -The  paper  prototype  is  tested 
"with  "typical”  users.  These  Risers  are  giventasks,  as  identified  in  the  needs  analysis,  and  asked  to  complete  these 
tasks.  Revisions  are  made  as  necessary,  and  the  paper  prototype  is  converted  to  a computer-based  prototype.  Again, 
usability  testing  is  conducted,  this  time  with  the  computer  prototype.  Once  again,  the  usability  testing  is  conducted 
with  users  being  asked  to  complete  predetermined  tasks. 

However,  not  all  visitors  to  a web  site  arrive  with  particular  questions  in  mind.  These  users  are  "browsing,” 
or  in  other  words  simply  visiting  the  site  to  see  what  it  has  to  offer.  They  may  return  later  with  particular  questions, 
but  their  first  visit  can  be  characterized  as  "just  looking."  Usability  testing  with  predetermined  tasks  can  provide 
valuable  clues  as  to  whether  the  site  meets  the  needs  of  the  information  provider  (is  the  site’s  content  logically  and 
intuitively  arranged,  so  that  "typical"  users'  questions  are  answered?).  However,  predetermined  tasks  do  not  address 
the  usability  issues  posed  by  visitors  who  are  "just  browsing." 

The  purpose  of  this  study  is  to  determine  whether  visitors  who  are  "just  browsing"  can  do  so  more 
effectively  under  one  of  the  three  menu  schemas  described  above  (hierarchical  disappearing  menus,  persistent  menus 
via  frames  or  tables,  and  representations  of  navigational  maps).  An  additional  research  question  pertains  to  the 
usability-testing  procedure  itself:  should  usability  testing  include  unstructured  tasks,  as  well  as  predetermined  tasks? 
In  this  study,  the  content  is  three  versions  of  the  top-level  structure  of  the  web  site  for  the  Indiana  University  School 
of  Education. 

This  study  examined  which  of  the  three  menu  designs  worked  better  for  a user  who  is  not  given  a particular 
task.  For  example,  after  using  a site  for  10  minutes  without  a specific  task,  what  would  be  the  user’s  conception  of 
the  structure  of  the  web  site? 

Literature  Review 

Boling  & Frick  (1998)  describe  an  approach  to  web  design  which  depends  largely  on  the  determination  of 
frequently-asked  questions  and  designing  usability  tasks  related  to  those  questions.  This  approach  is  built  upon  the 
fundamental  premise  that  users  are  looking  for  particular  information  and  have  predetermined  goals  when  accessing 
a web  site.  This  method  for  web  design  has  proven  successful  in  developing  information- intensive  sites  at  Indiana 
University.  One  shortcoming  of  this  approach  is  its  inability  to  incorporate  what  some  have  called  “browsing”  type 
behavior.  “A  major  source  of  hypertext  usability  problems  stem  from  the  fact  that  hypertext  systems  are  designed  to 
encourage  and  facilitate  browsing”  (Smith,  1997). 

Implications  of  Browsing  Behavior 

Browsing,  due  to  its  serendipitous  nature,  is  difficult  to  simulate.  When  truly  browsing  a site,  users  have 
their  own  goal  set,  which  may  or  may  not  be  obvious  to  the  researcher  and  which  may  change  over  time  as 
information  is  gathered  and  refined.  This  may  help  to  explain  the  findings  from  Remede,  Gomez  and  Landauer 
(1987)  which  suggest  that  the  main  cause  of  inability  to  find  information  in  a system  is  that  users  describe  what  they 
are  looking  for  in  different  terms  to  those  the  system  knows. 

Because  the  web  is  a relatively  recent  development,  most  research  studies  involving  the  web  have  been 
studies  provide  demographic  information  about  users  with  only  scanty  information  about  how  they  are  using  it 
(Catledge,  1995).  Recent  studies  by  Raman  (1996)  about  browsing  behavior  in  particular,  found  that  individuals 
enjoyed  the  feeling  of  exercising  control  over  what  they  wanted  to  see,  and  the  ability  to  tap  into  vast  amounts  of 
information  on  diverse  topics.  In  evaluating  a hypermedia  library  information  system,  Zhu  (1995)  found  that  users 
chose  browsing  much  more  often  than  querying  and  used  content-based  browsing  much  more  often  than  using  an 
index. 

Hendry  and  Harper  (1997)  discuss  how  opportunistic  searchers  (those  who  work  the  way  they  want  to,  as 
opposed  to  following  a determined  set  of  goals  or  strategies)  might  benefit  from  "simple  systems  that  create  a 
flexible  and  revealing  dialogue  between  the  system  and  user."  They  suggest  that  by  representing  search  plans  or 
navigational  routes  on  a page,  opportunistic  searchers  may  be  better  equipped  to  perform  informal  problem-solving. 
However,  usability  tests  which  rely  on  predetermined  tasks  do  not  support  the  opportunistic  searcher;  those  needs 
may  be  less-formally  defined. 

Attention  to  cognitive  organization  of  the  user  was  also  the  focus  of  study  in  Coll  (et  al.  1993).  This  study 
found  relevance  in  the  common  design  guideline  of  using  nine  or  fewer  menu  items,  in  addition  to  finding  that  good 
conceptual  layout  of  the  menu  on  screen  is  more  important  than  good  physical  layout. 

The  non-linear  way  in  which  we  learn,  and  this  similarity  with  the  structure  of  hypertext  was  tested  by 
Jannasch-Pennell  (1996).  This  study  investigated  the  impact  of  learner  control  on  performance  in  a hypertext 
instructional  environment.  The  study  examined  five  learner  levels  across  different  educational  levels  (middle  school, 
high  school,  undergraduate,  graduate).  Performance  was  measured  through  scoring  on  a criterion  measure.  Using 
covariance  analysis,  the  study  examined  the  performance  of  the  educational  levels  against  the  variables  of  computer 


confidence/competence,  level  of  statistical  knowledge,  experience  with  the  Web,  and  gender.  While  the  abstract 
identified  the  hypothesis  of  this  study  to  be  "the  educational  level  and  the  interface  characteristics  have  a direct 
impact  on  user  performance,"  no  mention  of  the  study’s  outcome  was  provided. 

Catledge  (1995)  points  out  that  most  usability  studies  that  have  been  conducted  are  generally  performed  in  a 
closed,  single-authored  system.  It  is  apparent  that  given  the  complexity,  size  and  ever  changing  nature  of  the  web, 
any  global  scale  usability  study  would  prove  less  than  fruitful.  Because  of  the  known  propensity  of  many  users  to 
browse,  additional  testing  measures  should  be  taken  to  simulate  such  behavior,  to  ensure  more  authentic 
generalizations  from  the  research  findings. 

Consistency  and  Persistence  in  Navigation 

Although  "pre-web"  in  its  origin,  a study  by  Lucas  (1991)  provides  several  recommendations  of  design  for 
computer-learner  interface,  mostly  based  on  previous  research  and  testing.  He  suggests  consistency  in  screen  layout, 
using  visual/graphical  images  to  enhance  textual  descriptions,  and  menu  selection  interfaces  that  prevent  the  need  for 
the  learner  to  learn  commands.  Consistency  is  also  a primary  concern  in  the  Apple  Web  Design  Guide  (1997), 
which  states  that  “providing  navigation  links  in  standard  places  helps  people  to  develop  a perception  of  stability  in 
your  site  and  makes  navigation  easier.”  Among  his  16  recommendations  on  web  page  design  Tillman  (1997)  makes 
these  navigation-related  prescriptions:  each  node/destination  be  able  to  stand  alone,  provide  visual  clues  indicating 
users'  selections  are  being  processed,  and  employ  labeled  and  unlabeled  links. 

Hardman  (1989)  used  observational  evidence  to  decipher  that  many  users  return  frequently  to  the  screen 
from  which  they  began,  which  may  suggest  that  users  desire  to  get  back  to  a known  place  instead  of  working  out  a 
route  which  will  take  them  from  where  they  are  to  where  they  want  to  go.  Additional  credibility  is  given  to  this 
finding  through  a study  at  the  Georgia  Institute  of  Technology  involving  107  users  over  a three  week  period  in  which 
using  the  browser’s  “Back”  command  or  button  accounted  for  a full  41%  of  all  navigation. 

Catledge’s  study  at  GIT  revealed  that  the  typical  user  accessed  on  average  10  pages  per  server,  which 
would  lead  to  the  conclusion  that  the  “must  see”  information  should  be  accessible  within  two  or  three  jumps  of  the 
initial  home  page  (two/three  navigation’s  in,  two/three  out,  performed  two/three  times)  a finding  also  suggested  by 
Apple. 

Other  relevant  studies  in  persistent  navigation  include,  Smith  (1994)  where  testing  was  done  using  both 
two-and  three-dimensional,  spatial  and  schematic  navigational  aids,  as  well  as  Andrews  (1996)  where  the  use  of  a 
two-dimensional  structure  map  which  visualizes  link  and  membership  relationships,  with  a three-dimensional 
landscape  visualization,  presenting  a combined  display  of  both  hierarchical  structure  and  hyperlink  relationships 
helped  to  prevent  users  from  losing  their  orientation. 

Utilizing  Mental  Maps  in  Information  Recall 

In  her  doctoral  dissertation,  Tyan  (1989)  showed  that  a top  level  structure  awareness  greatly  enhanced 
readers  recall  performance.  This  study  was  done  in  part  by  having  readers  complete  a schematic  diagram  of  a 
passage  previously  read,  and  found  a positive  correlation  between  knowledge  of  structure  and  recall  of  information 
within  the  structure.  Although  this  study  was  done  with  print-based  text,  the  principles  of  recall  may  apply  to  web 
based  text  as  well. 

Increasingly,  researchers  and  design  practitioners  are  calling  for  the  integration  of  spatial  metaphors  when 
designing  Web  pages.  Dieberger  and  Bolter  (1995)  argue  the  Web  is  inherently  geographical,  leading  users  to 
organize  their  search  strategies  in  spatial  forms.  And  Heid  (1996)  suggests  image  maps  actually  help  improve 
navigation  by  allowing  developers  to  include  backward  compatibility  with  forward-looking  design. 

In  one  study,  Philleo  asked  a group  of  preservice  teachers,  for  whom  the  Internet  was  a relatively-new  area 
of  study,  to  draw  "...a  diagram  or  picture  of  the  Internet  as  you  understand  it"  (1995).  Philleo  concluded  that  "it  may 
be  that  in  order  to  solve  problems  that  arise  with  Internet  use  or  to  be  fully  empowered  to  explore  its  resources,  some 
sort  of  realistic-looking  mental  image  might  prove  helpful."  Taking  Philleo’s  findings  further  - or  perhaps,  by  scaling 
it  down  to  the  scope  of  a single  web  site  - it  may  be  that  a browsing  user's  pictorial  image  may  provide  information 
about  what  the  user  thinks  the  site  contains  (see  also  Apple,  1997).  Beyond  this,  by  providing  a persistent  site  map, 
one  which  shows  how  a user  arrived  at  a particular  page,  a user  may  be  helped  in  maintaining  a mental  image  of  the 
site,  making  subsequent  visits  more  fruitful  and  productive. 

In  a similar  study  to  this  proposal,  Chang  (1995)  used  three  types  of  hypertext  menu  designs  while  testing 
against  individual  difference  variables.  The  three  menu  styles  were  1)  hierarchical  structure  signaled  by  explicit 
menus  and  referential  links  presented  by  embedded  menus,  2)  hierarchical  structure  signaled  by  embedded  menus 
and  referential  links  presented  by  embedded  menus,  and  3)  referential  links  presented  by  embedded  menus  without 
structural  information.  The  two  individual  variables  were  cognitive  style  (field-dependence  versus  field- 
independence)  and  epistemic  beliefs  and  preferences  (simple  versus  complex).  The  study  population  of  52  students 
at  the  University  of  Illinois  yielded  the  following  conclusions:  Cognitive  style  was  found  to  an  influential  factor  on 
searching  performance  as  field-independent  users  fared  far  better,  especially  when  no  structural  information  was 

O 

ERIC 


113 

110 


provided;  Users'  beliefs  about  whether  knowledge  is  simple  or  complex  did  not  affect  searching  performance; 
Program  #1  proved  to  be  the  most  efficient,  but  did  not  always  produce  the  best  search  scores;  and  most  novice 
subjects  given  simple  search  tasks  chose  linear  links  over  non-linear  hypertext. 

In  summary,  the  literature  indicates  that: 

1.  Current  research  shows  that  people  use  various  types  of  browsing  behavior  in  navigating  through  web  sites,  and  such 
behavior  is  difficult  to  simulate  in  research  activities. 

2.  Consistency  in  screen  layout  and  design  assists  users  in  navigation  through  a particular  web  site. 

3.  Users  tend  to  return  frequently  to  the  “top  level”  or  screen  from  which  they  first  entered  a site  rather  than  working  out 
paths  which  will  take  them  where  they  want  to  go. 

4.  A mental  awareness  of  the  structure  that  information  is  presented  in  aides  in  the  recall  of  the  material. 

Methodology 

The  comparative  inquiry  of  the  three  treatments  used  volunteer  graduate  and  undergraduate  subjects  from 
Indiana  University  Bloomington.  As  subjects  were  recruited,  they  were  assigned  at  random  to  one  of  the  three 
treatments  (total  N = 44).  In  keeping  with  University  policy  concerning  the  use  of  human  subjects,  participants  were 
given  a copy  of  an  Informed  Consent  document,  which  explained  the  nature  of  the  study  and  assured  anonymity. 

After  reviewing  and  signing  the  Informed  Consent  document,  the  subjects  were  asked  to  complete  a brief 
questionnaire  to  determine  the  their  level  of  familiarity  with  the  current  web  site. 

Because  many  of  the  hypertext  links  within  this  site  were  originally  designed  to  take  the  user  to  other  web 
sites,  and  because  our  research  questions  concerned  only  the  navigability  of  the  site  in  question,  off-site  links  were 
changed  so  that  they  would  produce  an  error  message  (normally,  this  would  produce  a ”404  not  found”  error 
message;  the  server  was  reconfigured  to  deliver  a more  user-friendly  message).  The  error  message  was  pre-set  to 
describe  the  reason  the  document  was  missing,  and  the  subjects  were  told  that  finding  one  of  these  pages  was  a 
possibility;  that  if  they  did,  they  should  use  the  browser’s  back  button  to  return  to  the  previous  page  and  continue 
browsing. 

The  subjects  were  then  shown  the  home  page  for  the  treatment  they  were  assigned  to;  they  were  told  that 
their  assignment  was  to  simply  browse  the  site  for  a period  of  time  and  get  an  idea  of  its  content.  The  three 
treatment  conditions  can  be  viewed  on  the  Web  at  URL:  http://education.indiana.edu/-frick/aect99/designs.html  (A  = 
non-persistent  (changing)  hierarchical  menus  displayed  as  vertical  lists  of  choices  with  non-hot  exemplars  to  the 
right;  B = A + persistent  top-level  menu  displayed  as  a vertical  list  of  choices  in  the  left  column  of  the  screen;  C = A 
+ horizontal  hierarchical  lists  of  choices  at  the  top  of  the  screen  that  represent  the  path  taken  down  the  hierarchy 
[textual  map] ). 

The  subjects  then  proceeded  to  browse  the  site  while  the  researchers  observed  and  recorded;  the  starting 
time,  number  of  "back  button”  clicks,  and  subject  comments  were  the  primary  observations  made.  Browser  back 
button  clicks  were  recorded  as  having  been  "forced”  or  "unforced.”  Forced  back  button  clicks  were  the  result  of 
having  arrived  at  a "document  not  found”  page  (as  described  above);  unforced  back  button  clicks  were  all  others. 

After  browsing  the  web  site  for  about  ten  minutes,  the  researcher  asked  the  subject  to  stop.  A questionnaire 
was  then  completed  which  gathered  general  information  about  the  subject  and  their  opinions  about  certain  aspects  of 
the  site.  General  information  included  native  language,  major,  and  various  aspects  of  computer  use;  opinions  about 
the  web  site  concerned  navigability,  affective  response,  and  understanding  of  the  purpose  and  navigational  structure 
of  the  web  site.  The  subjects  were  also  asked  four  open-ended  questions  regarding  what  they  liked  or  did  not  like 
about  the  web  site. 

In  the  final  aspect  of  the  session,  the  subjects  were  asked  to  complete  a conceptual  map  of  the  web  site. 
Because  recall  of  the  material  may  be  a factor  in  the  subjects'  conceptual  understanding  of  the  site,  it  was  decided 
that  at  least  ten  minutes  should  elapse  between  the  end  of  the  browsing  activity  and  the  beginning  of  the  conceptual 
map  test;  if  the  subject  completed  the  questionnaire  in  less  time,  an  unrelated  activity  was  begun  to  take  up  the 
remainder  of  the  time. 

Analysis 

Following  the  data  collection  process  described  in  the  methodology,  each  subject’s  responses  were  reviewed 
and  analyzed.  The  data  yielded  through  the  testing  included  (1)  overall  browsing  time,  (2)  total  unique  pages  visited, 
(3)  conceptual  map  score,  (4)  the  number  of  times  the  "back”  button  was  used,  and  (5)  ancillary  data  such  as 
comments  made  during  testing  and  written  responses  on  the  post-test  questionnaire.  The  web  page  logs  were  also 
retrieved  for  each  subject's  testing  period  to  verify  the  team  member’s  observations. 

Conceptual  map  tests  were  first  scored  independently  by  two  persons,  then  the  average  of  two  was 
calculated.  However,  we  found  the  correlation  between  two  persons’  scores  was  not  sufficient  (0.74).  In  order  to 
improve  the  reliability  of  scoring,  two  scorers  scored  each  conceptual  map  together  by  reaching  a consensus  score 
for  each.  Then,  three  tests  from  each  design  were  chosen  randomly  and  a blind  observer  scored  them  without 
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knowing  which  design  they  were.  The  blind  observers’  scores  were  then  correlated  with  the  consensus  scores,  but 
again  the  correlation  was  low  (0.68). 

In  the  statistical  analysis,  the  recorded  variables  were  coded  into  an  SPSS  file  and  subjected  to  a number  of 
statistical  manipulations.  In  particular,  an  attempt  to  correlate  total  browsing  time  and  conceptual  map  scores,  as 
well  as  total  unique  pages  visited  and  total  browsing  time  was  made  through  correlation  (Pearson  r)  and  a one-way 
analysis  of  variance  (ANOVA). 

Unfortunately,  the  data  collected  on  back  button  usage  was  not  usable.  This  occurred  for  a number  of 
reasons.  Observers  attempted  to  record  usage  of  the  back  button  as  an  "ordinary”  choice  of  the  user  compared  to  a 
"forced"  choice.  Since  all  pages  in  the  designs  that  led  to  external  links  under  normal  conditions  were  closed,  users 
were  forced  back  to  a previous  page.  Due  to  an  oversight  on  the  research  procedures,  not  all  observers  distinguished 
between  these  two  uses  of  the  back  button.  Furthermore,  the  web  logs  could  not  accurately  verify  the  types  of  back 
button  usage  as  had  been  anticipated,  so  the  data  was  discarded. 

Research  Findings 

An  ANOVA  was  undertaken  to  see  if  there  were  any  statistically  significant  differences  among  three 
different  designs.  Average  conceptual  map  scores  in  the  three  designs  were  nearly  identical. 

Table  1.  Means  and  Standard  Deviations  for  Conceptual  Map  Scores  in  the  Three  Menu  Designs 


Design 

Mean 

Standard  Deviation 

A (traditional  non-persistent  hierarchical  menu)  n=14 

3.43 

2.62 

B (persistent  main  menu  — left  side)  n=15 

3.87 

1.92 

C (persistent  textual  map  - top  left-to-right)  n=15 

3.27 

2.31 

There  were  no  statistically  significant  differences  among  the  three  groups  ( p < .05)  with  respect  to  subjects’ 
conceptual  understanding  of  the  overall  structure  of  the  Web  site.  Based  on  both  the  statistical  analysis  and  the 
interpretation  of  test  subject’s  written  and  verbal  comments,  few  subjects  noticed  the  differences  between  the  three 
test  designs.  Observers  noted  that  subjects  tended  to  use  the  non-persistent  links  for  navigation  that  were  present  in 
all  three  designs.  Most  subjects  did  not  use  the  persistent  menu  on  the  left  (sidebar)  or  the  textual  chain  (top)  for 
navigation  while  browsing,  since  they  apparently  did  not  "see”  them.  Moreover,  these  conceptual  map  scores  were 
quite  low  with  little  variation  among  subjects  within  groups.  Not  only  did  this  limit  reliability  of  the  measure  and 
hence  the  possibility  of  significant  differences,  but  it  also  indicated  poor  conceptual  understanding  of  the  Web  site 
(lack  of  an  accurate  mental  model). 

Pearson  product-moment  correlation’s  were  run  between  demographic  and  other  variables  and  conceptual 
map  scores.  While  most  of  the  correlation  values  were  not  significant,  there  were  statistically  significant 
correlation’s  between  conceptual  map  scores  and  how  often  subjects  used  web  the  web  per  week  (0.319)  and  the 
number  of  software  products  used  frequently  (0.338).  These  findings  indicate  those  subjects  who  used  the  Web  and 
computer  software  products  more  frequently  were  better  able  to  grasp  the  conceptual  structure  of  a new  Web  site  in  a 
relatively  short  period  of  time. 

Conclusion  and  Reflections 

While  this  study  did  not  discover  a design  format  that  convincingly  facilitates  browsing  behavior  over  other 
formats,  as  the  study  had  intended,  there  are  several  possible  interpretations,  which  will  require  further  research 
before  any  firm  conclusions  can  be  reached.  One  interpretation  is  that  subjects  tend  not  to  form  mental  models  of 
Web  site  structures,  which  would  explain  the  low  conceptual  map  scores  overall  and  lack  of  differences  among  the 
three  menu  formats.  Spool  et  al.  (1997)  conclude  from  their  experiences  in  usability  testing  that  most  people  do  not 
form  mental  models  of  Web  sites.  Another  interpretation  is  that  there  are  no  differences  in  conceptual  understanding 
that  can  be  attributed  to  the  three  types  of  menu  designs  that  were  studied.  Evidence  to  support  this  are  observations 
that  most  users  did  not  use  those  persistent  navigation  features  because  they  did  not  notice  them.  If  so,  then  we 
would  not  expect  differences  among  the  designs,  since  there  were  none  from  the  user’s  perspective  What  does 
appear  to  be  supported  by  findings  in  this  study  is  that  more  experienced  computer  users  do  seem  to  achieve  a 
greater  conceptual  grasp  of  a Web  site’s  structure  when  browsing  over  a relatively  short  period  of  time.  The  number 
of  software  products  used  frequently  indicates  greater  computer  experience,  as  well  as  more  frequent  use  of  the  Web. 

A number  of  points  should  be  considered  on  how  this  study  was  conducted  and  how  a similar  future  study 
might  do  things  differently.  As  previously  mentioned,  tracking  usage  of  the  back  button  proved  difficult  in  this 
study.  In  general,  the  speed  at  which  subjects  browsed  through  the  pages  indicates  that  a more  efficient  method  of 
accurately  following  browsers’  paths  needs  to  be  devised.  While  Web  logs  are  useful  to  verify  the  pages  a subject 
visits,  they  do  not  reveal  if  she  or  he  uses  the  back  button,  or  specially  designed  navigation  buttons  like  the  ones 
included  in  designs  B and  C as  alternative  shortcuts  to  previously  visited  pages.  Moreover,  browsers  cache  Web 
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pages  recently  visited,  and  no  Web  server  log  entries  are  made  when  they  go  back  to  those  locally  cached  pages. 
Videotaping  subjects'  use  of  the  Web  site  (or  capturing  with  a mediator  and  recording  on  video)  would  allow 
observers  afterwards  to  more  accurately  track  rapid  navigation  by  users  and  the  means  by  which  they  navigate. 

Future  studies  should  also  consider  the  type  of  test  subjects  employed.  This  study  was  limited  by  its  use  of 
mostly  undergraduate  students,  many  of  whom  were  only  first  or  second  year  students.  Since  the  awarding  of  extra- 
credit was  necessary  to  leverage  an  adequate  supply  of  test  subjects,  the  motivation  of  a large  proportion  of  the  test 
population,  and  thus  the  results  of  their  test  sessions  could  be  questioned.  Perhaps  future  studies  should  consider 
using  a different  population,  such  as  professional  staff  of  a school  or  department.  This  would  also  allow  extending 
the  testing  period  over  a longer  period,  rather  than  just  the  average  10  to  30  minutes  experienced  with  the  student 
population.  Indeed,  an  expanded  test  period  of,  for  example,  two  days  a week  for  30  minutes  over  the  course  of 
several  weeks  would  most  likely  yield  more  conclusive  results  than  those  found  here. 

The  last  observation  for  potential  future  studies  relates  to  measuring  a users  structural  understanding  of  a 
web  site.  This  study  measured  comprehension  through  the  completion  of  a conceptual  map.  As  cited  in  the  literature 
review,  there  is  evidence  that  users  form  their  understanding  of  web  page  arrangements  in  spatial  patterns.  While  the 
methodology  used  here  was  in  accordance  with  previous  studies,  future  studies  may  consider  using  an  alternative 
measurement  tool. 

In  the  case  of  this  study,  a more  effective  alternative  method  may  have  been  to  ask  the  user,  after  browsing 
the  site  for  a period  of  time,  to  list  what  kind  of  answers  this  site  could  provide  to  them.  While  this  approach  would 
place  greater  responsibility  on  the  researcher  to  interpret  user's  responses,  it  would  eliminate  the  constrictions  users 
faced  in  this  study  in  terms  of  the  way  in  which  they  were  required  to  recall  multiple  first-  and  second-level  headings. 
As  noted  earlier,  the  speed  at  which  many  of  the  subjects  moved  through  the  test  sites  may  have  inhibited  their  ability 
to  form  a spatial  interpretation  of  the  site.  While  the  speed  of  navigation  witnessed  in  this  study  suggests  the  designs 
presented  the  information  efficiently,  this  efficiency  had  an  unintended  negative  impact  on  comprehension  and  recall. 

One  final  thought  on  this  study  addresses  the  very  nature  of  web  browsing  and  the  inherent  difficulty  in 
measuring  it.  Based  on  the  observation  of  the  test  subjects,  a question  emerged  related  to  the  way  the  information  in 
the  design  sites  was  presented.  This  question,  which  also  applies  to  other  web  sites,  is  whether  the  manner  in  which 
information  is  presented  creates  self-forming  goals  within  the  user.  Through  observations  made  in  this  study,  many 
test  subjects  "found"  a topic  or  point  of  interest,  which  prompted  them  to  continue  searching  for  additional 
information  related  to  their  discovery.  For  example,  when  coming  across  the  faculty  and  staff  directory,  the  subjects 
began  searching  for  one  particular  individual,  usually  a professor  whose  class  they  were  enrolled  in.  This  led  them 
search  that  individual's  personal  page,  or  look  for  their  office  in  the  School  of  Education  building  map.  When  these 
subjects  "found"  something  of  interest  and  began  pursuing  addition  related  information,  they  were  no  longer 
browsing,  but  they  were  now  driven  by  a task. 
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Abstract 

This  study  investigated  performance  differences  between  three  different  web-based  navigation  models: 
linear ; persistent ; and  semi-persistent  menu  structures.  Forty-four  college  students  were  placed  into  one  of 
the  three  navigation  conditions  and  completed  information-finding  tasks.  No  significant  differences  were 
found  among  the  models  with  respect  to  success  in  completing  tasks  or  overall  completion  time.  Use  of 
other  navigation  aids  built  into  the  browser  such  as  the  HOME  and  BACK  buttons  was  also  measured,  as 
well  user  satisfaction  and  perceived  usability.  No  differences  among  models  were  found  here  either. 
Results  of  this  study  tentatively  indicate  that  the  three  navigation  models ; as  tested,  performed  equally  well 
and  further  that  designers  should  perhaps  give  fair  consideration  to  each  of  the  models  pending  the 
purpose  and  audience  of  the  site  being  designed.  Further  research  is  needed,  since  several  potentially 
confounding  factors  were  identified. 

Introduction 

Everyday  thousands  of  people  around  the  world  make  it  a part  of  their  day  to  locate  some  piece  of 
information  on  the  World  Wide  Web  (WWW).  Sometimes  users  of  the  WWW  are  browsing,  simply  looking  at 
various  web  sites  to  find  something  that  interests  them,  while  other  users  access  the  WWW  with  a specific  task  in 
mind. 

Web  sites  vary  a great  deal  in  terms  of  the  audience  being  served  and  the  purpose  of  the  content  being 
provided.  There  are  sites  that  target  children,  adolescents  and  adults;  educators,  business  executives,  and 
researchers;  vacationers,  television  junkies,  and  sports  fanatics.  The  list  goes  on  as  long  as  there  are  topics  and 
people  who  are  interested  in  them.  Of  all  these  sites  that  are  currently  in  use,  some  are  intended  to  be  used 
occasionally  or  even  rarely,  while  there  are  other  sites,  such  as  corporate  Intranet  sites,  that  are  used  daily  for  very 
specific,  job-critical  tasks. 

Regardless  of  the  usage  context  for  a particular  web  site,  all  web  sites  must  employ  some  kind  of  navigation 
scheme  to  aid  the  user  in  traveling  from  one  page  to  another.  Designers  of  web  sites  currently  enjoy  little  in  the  way 
of  published,  empirically  supported  guidelines  specific  to  navigation  issues.  Site  navigation  is  an  essential  element 
of  an  effective  web  site  and  there  is  a pressing  need  for  researchers  to  begin  investigating  the  issues  and  questions 
surrounding  web-based  navigation  models.  Most  sites  today  are  still  built  based  either  on  personal  experience  and 
taste,  an  incompatible  medium  (i.e.,  paper),  or  at  best  on  usability  data  with  limited  generalizability. 

The  purpose  of  this  study  was  to  investigate  the  effectiveness  of  three  popular  navigational  models  used  in 
on  the  WWW.  Specifically  this  study  had  hoped  to  begin  laying  the  foundation  for  future  studies  dealing  with  web 
sites  and  how  people  perform  using  them. 

Review  of  Related  Literature 

Though  the  research  base  on  navigating  within  a WWW  environment  is  limited,  there  are  a number  of 
studies  which  have  been  done  in  the  last  twenty  years  that  have  important  implications  for  designing  navigation 
schemes  for  web-based  information  systems.  The  varying  research  areas  of  influence  include  studies  done  on 
paper-based  navigation,  pre-web  hypertext  models,  and  web  site  usability.  Hierarchical  and  persistent  menu 
strategies  on  the  web  are  a relatively  new  area  of  interface  design  interest.  Only  recently  have  researchers  begun  to 
address  the  question  of  menu  development  as  applied  to  a hypertext  environment,  and  in  particular  the  World  Wide 
Web.  Empirical  research  is  just  beginning  to  emerge,  mostly  in  the  form  of  dissertations  and  research-oriented 
courses  and  seminars. 
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Effectiveness  and  Challenges  of  Hierarchical  Information  Structures 

Research  done  by  Chang  (1995)  indicates  that  hierarchical  structures  result  in  the  most  efficient  navigation 
and  information  retrieval.  In  designing  specific  hierarchical  menu  structures,  depth  (the  number  of  levels)  and 
breadth  (the  number  of  items  per  page)  were  shown  to  be  prime  considerations  by  Harney  (1990).  Other  factors  that 
have  been  shown  to  be  of  importance  in  hierarchical  menu  design  include: 

1.  menu  dimension, 

2.  task  complexity,  and 

3.  the  knowledge  structure  of  the  WWW  user  (Jacko,  1994). 

It  has  been  shown  that  WWW  users  experience  difficulty  navigating  large  information  systems.  McKnight, 
Dillon,  and  Richardson  (1990)  provided  evidence  that  users  can  become  disoriented  during  hypertext  searching 
tasks.  In  this  study,  as  in  many  information  spaces,  the  challenge  for  the  user  is  to  know  where  they  are  in  the 
system,  where  they  want  or  need  to  go  to  in  the  system,  and  finally  to  somehow  keep  track  of  where  they  have  been 
in  the  system.  Disorientation  results  when  users  can  not  accomplish  those  tasks.  It  has  been  proposed  that  hypertext 
designs  can  be  improved  by  providing  structural  cues  to  the  WWW  user,  thereby  facilitating  user  orientation  (Rouet, 
et  al.  1996). 

One  of  the  few  recommendations  one  can  locate  in  the  research  comes  from  Sano  (1997)  where  he  states 
that  an  effective  and  intuitive  online  information  structure  has  two  critical  characteristics.  First,  the  information 
which  users  need  the  most  should  be  within  two  to  three  levels  away  from  their  starting  point.  Second,  grouping  of 
related  items  is  essential  to  avoiding  long  lists  of  information  that  the  user  must  scroll  through. 

Structural  Cues 

Paper  documents  have  long  used  a wide  array  of  navigation  aids  to  help  users  find  the  information  they 
need.  Tables  of  contents  have  been  shown  to  be  effective  navigation  aids  in  paper  documents.  Indices,  vertical  tabs, 
and  other  devices  have  also  been  used  effectively  in  paper-based  documents. 

Many  web-based  information  systems  continue  to  use  individual  vertical  lists  of  links  for  each  level  in  a 
hierarchy.  Frick,  et  al.  (1995)  and  Corry,  et  al.  (1997)  found  that  these  lists  worked  well  in  meeting  user 
expectations  and  resulted  in  high  navigation  performance  on  a university  web  site.  In  the  same  design,  exemplars 
were  added  to  each  link  to  aid  the  user  in  understanding  more  about  what  each  link  would  really  provide  them. 

Users  in  this  study  reported  that  these  exemplars  were  helpful.  Spool  (1997)  also  reports  that  link 
descriptors  have  been  shown  in  usability  testing  to  be  extremely  effective  aids  in  helping  users  to  decide  whether  the 
link  is  in  fact  what  they  want  or  not. 

Another  navigation  tool  used  extensively  in  online  systems  is  the  expanding/contracting  hierarchy.  Such 
lists  of  choices  have  been  shown  to  be  effective  in  the  Windows  operating  system,  as  well  as  in  the  Apple  Computer 
OS.  Early  hypertext  systems  such  as  Superbook  have  also  been  shown  to  use  expanding/contracting  outlines 
successfully  (Egan,  et  al.  1989).  Specific  to  the  web,  Nation,  et  al.  (1997)  conducted  an  evaluation  of  an 
expandable/contractible  table  of  contents  tool  called  WebTOC.  The  tool  is  a Java  applet  that  displays  a dynamic 
hierarchy  on  the  left  of  the  screen  with  the  content  to  the  right  side.  Early  results  show  that  the  applet  can  assist  users 
in  navigating  web  sites.  Lastly,  expanding/contracting  tables  of  contents  have  also  been  shown  to  decrease  browsing 
times  when  compared  with  fully  expanded  versions  (Chimera  et  al.  1994). 

Another  navigation  aid  that  is  popular  on  the  web  is  what  we  call  hierarchical  chains.  These  chains  are 
structural  cues  in  that  they  communicate  to  the  user  exactly  where  in  the  hierarchy  they  are  and  where  they  are 
currently.  Usually  a horizontal  line  of  hypertext  items  separated  by  colons  or  arrows,  each  item  in  the  chain  is  a level 
in  the  hierarchy  that  is  hot-linked,  allowing  the  user  to  return  with  a single  click  to  any  previously  visited  level. 
Examples  of  this  navigation  strategy  can  be  seen  at  two  popular  and  heavily  usability-tested  web  sites.  They  are: 

http://www.yahoo.com 

http://www.useit.com. 

Nielsen  has  usability  tested  this  method  and  found  it  to  be  an  effective  aid  in  helping  users  navigate 
web-based  information  systems. 

Additional  Design  Considerations 

After  determining  the  overall  hierarchical  design,  other  issues  that  need  to  be  addressed  concern  types  of 
menu  structure  (embedded  versus  explicit)  and  sub-menu  design.  Some  research  indicates  that  embedded  menus 
produce  more  accurate  searches,  permit  fewer  screens  viewed,  and  are  preferred  overall  by  users  (Koved  & 
Shneiderman,  1986).  Menu  design  can  also  be  divided  into  three  levels:  alphabetical,  categorical,  and  random.  In 


terms  of  overall  efficiency  for  novice  users,  there  is  no  significant  difference  between  alphabetical  and  categorical 
(Coll,  Coll,  and  Nandavar,  1993). 

Another  consideration  for  web  designers  concerns  the  type  of  tasks  that  need  to  be  supported.  Navigation 
aids  in  a hypertext  environment  should  be  developed  according  to  the  tasks  of  the  user.  In  terms  of  information 
search  tasks,  navigation  via  indexes  seems  to  be  the  preferred  method  of  users  (Edwards  and  Hardman,  1989,  cited  in 
Smith,  1997).  Developers  should  also  provide  information  to  the  user  informing  them  of  their  current  progress,  such 
as  a path  history  (Jones,  Farquhar  and  Surry,  1995).  In  terms  of  sub-menu  design,  studies  by  Scheurman  and  Peck 
(1991)  indicate  that  users  of  sub-menus  that  provide  a return  to  main  menu  option,  versus  a return  to  previous 
sub-menu  option,  spend  less  time  on  sub-menus  and  therefore  are  more  efficient. 

Other  issues  of  importance  that  need  to  be  considered  are  usability  and  its  measurement,  which  have 
included  various  factors  in  the  past  such  as:  time  to  completion,  number  of  errors,  and  deviations  from  optimal  paths 
(Smith,  1994,  cited  in  Smith,  1997).  Other  considerations  could  include  finding  required  information,  identifying  the 
meaning  of  a link,  and  obtaining  the  required  information  when  it  is  requested  (Smith,  1997). 

There  are  currently  a number  of  different  navigation  strategies  that  designers  implement  in  web-based 
information  systems.  Some  of  the  many  navigation  options  available  to  web  designers  include: 

■ plain  hypertext,  often  in  the  form  of  lists,  categories,  hierarchies  (e.g.  http://www.indiana.edu/iub/) 

■ frames  that  provide  persistent  or  perpetual  navigation  (e.g.  http://www.macromedia.com  and  http://www.steelcase.com) 

■ expandable/collapsible  hierarchies  (Danny  Goodman  s site) 

■ forms,  that  scroll  and/or  drop  down  lists  of  linkable  items  (e.g.  http://www.pw.com) 

■ hierarchical  chains  (e.g.  http://www.yahoo.com,  and  http://www.useit.com) 

Some  of  the  navigation  methods  listed  above  have  been  tested  with  users  both  in  usability  settings  and  in 
empirical  studies.  We  have  described  the  outcomes  of  those  efforts.  Other  methods  have  yet  to  be  tested  at  all,  or  at 
least  in  any  published  format. 

Methods 
Research  Design 

The  methodology  used  in  this  study  was  a classical  experimental  design  with  subjects  randomly  assigned  to 
one  of  three  treatment  conditions.  These  three  treatments  were  three  web  sites,  which  contained  same  information 
but  different  navigation  menus. 

■ Condition  A:  non-persistent  (changing)  hierarchical  menus  displayed  as  vertical  lists  of  choices  with  non-hot  exemplars  to 
the  right, 

■ Condition  B:  A + persistent  top-level  menu  displayed  as  vertical  list  of  choices  in  the  left  column  of  the  screen,  and 

■ Condition  C:  A + semi -persistent  horizontal  hierarchical  lists  of  choices  at  the  top  of  the  screen  that  represent  the  path 
taken  down  the  hierarchy  (textual  map). 

These  navigation  models  can  be  viewed  at  URL:  http://education.indiana.edu/~frick/aect99/designs.html. 

The  dependent  variables  in  this  study  were  the  subjects  performance  on  the  task  component  of  the  study,  the 
number  of  times  the  user  voluntarily  used  the  back  button,  and  each  subject’s  attitude.  Attitude  variables  were 
constructed  to  collect  subjective  data  about  (a)  appeal  of  the  designs,  (b)  their  usability,  and  (c)  perceptions  of 
"getting  lost"  in  the  web  sites. 

Pilot  Tests 

Six  pilot  tests  were  completed  prior  to  actual  data  collection.  The  pilot  tests  were  used  to  determine  the 
usability  of  data  collection  instruments,  to  determine  the  difficulty  of  the  tasks,  and  to  ensure  that  data  collection 
procedures  worked  satisfactorily. 

Instruments  tested  during  pilot  testing  included  a background  questionnaire,  task  list,  and  the  attitude 
survey.  The  results  of  the  pilot  tests  were  used  to  ensure  the  background  and  the  attitude  questionnaires  were 
understandable  and  usable.  The  pilot  test  was  also  used  to  finalize  the  task  list.  Over  30  tasks  were  tested  during  the 
pilot  tests.  The  difficulty  and  usability  of  each  task  was  determined  based  on  the  whether  the  subjects  found  the 
correct  answer  and  the  time  it  took  to  find  the  answer.  The  easiest  tasks  were  deleted  from  the  task  list,  and  24  tasks 
were  selected  for  use  in  the  actual  study. 

Instruments 

The  background  questionnaire  consisted  of  five  questions  about  the  subjects  web  and  computer  experiences. 
The  task  instrument  consisted  of  24  items.  The  task  items  were  standard  usability  tasks  that  users  were  asked  in 
order  to  find  specific  information  in  the  web  site.  The  tasks  were  evenly  distributed  among  the  web  site.  The  attitude 
questionnaire  was  an  instrument  consisting  of  18  Likert-type  questions  and  one  open-ended  question  for  general 
comments.  The  18  items  consisted  of  positively  and  negatively  worded  statements  about  the  navigation  model  used. 


which  were  intended  to  measure  its:  (a)  appeal,  (b)  usability,  and  (c)  "getting  lost"  issues.  In  response  to  each 
statement,  participants  indicated  which  one  of  five  ordered  responses,  ranging  from  Astrongly  agree  @ to  Astrongly 
disagree®. 

Prior  to  individual  sessions  with  each  subject,  the  researcher  needed  to  set  up  the  computer  and  browser  for 
the  test.  The  monitor  screen  resolution  was  set  to  640  X 480.  The  browser  was  opened  and  maximized.  The 
Adirectory  buttons®  were  disabled.  The  URL  was  set  to  the  treatment  tested  in  that  session.  The  AHOME®  button 
on  the  navigation  bar  was  set  to  the  Home  Page  URL  of  the  treatment.  The  Amemory  cache®  and  Adisk  cache® 
were  set  to  zero  so  that  web  server  logs  would  contain  navigation  moves  from  page  to  page.  The  IP  number  of  the 
computer  workstation  was  recorded  for  later  web  server  log  analysis.  Finally,  the  link  history  list  was  reset. 

Subjects 

Potential  participants  were  screened  beforehand  to  help  determine  who  would  be  appropriate  for  the 
research  study.  The  major  criterion  that  was  used  to  select  subjects  was  little  or  no  prior  experience  with  the  Indiana 
University  School  of  Education  web  site.  Subjects  were  excluded  who  were  already  familiar  with  this  web  site,  since 
prior  knowledge  might  confound  the  results.  The  resulting  44  subjects  were  undergraduate  and  graduate  students 
and  public  school  teachers,  who  were  randomly  assigned  to  the  three  treatment  conditions  ( n = 14,  15,  15 
respectively). 

After  completing  the  background  questionnaire,  each  subject  was  asked  to  find  answers  to  the  24 
information  tasks  (e.g.,  What  is  the  phone  number  of  the  External  Relations  Office?  Find  the  Web  page  for  the 
Education  Library.  Find  information  pertaining  to  Graduate  Admissions.).  The  tasks  were  randomly  ordered  for 
each  subject,  who  was  given  up  to  three  minutes  to  find  the  answer.  When  the  subject  believed  she  had  found  the 
answer,  she  notified  the  researcher.  If  the  answer  was  not  found  within  three  minutes,  the  subject  gave  up,  or  if  the 
subject  came  to  the  wrong  place  in  the  web  site,  it  was  counted  as  an  incorrect  response.  The  researcher  also 
recorded  task  completion  time,  and  navigation  buttons  used. 

Immediately  following  the  completion  of  the  24  task  questions,  each  subject  was  asked  to  complete  the 
attitude  questionnaire.  Most  subjects  completed  the  experiment  in  45  minutes  to  an  hour. 

Results 


Reliabilities  of  the  attitude  scales  were  estimated  by  Cronbach’s  alpha  for  appeal  (0.90),  usability  (0.89)  and 
getting  lost  (0.70).  The  number  of  completed  tasks,  the  total  time  for  completing  the  tasks,  the  total  number  of 
voluntary  back  button  clicks,  and  the  attitude  scales  were  analyzed  by  using  the  one-way  Avon’s.  The  statistical 
significance  level  used  for  hypothesis  testing  was  set  a priori  to  .05. 

There  was  no  significant  difference  among  group  means  on  number  of  tasks  completed  successfully. 


Table  1.  AN  OVA  for  the  number  of  tasks  completed  successfully 


Design 

Mean 

SD 

F 

Sig. 

A (n=  14) 

19.29 

2.16 

.069 

.934 

B (n  = 15) 

19.20 

2.43 

C(n=  15) 

18.93 

3.30 

There  was  no  significant  difference  among  the  three  designs  on  the  total  amount  of  time  to  do  all  24  tasks. 
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Table  2.  AN  OVA  for  the  total  task  completion  time  in  seconds. 


Design 

Mean 

SD 

F 

Sig. 

A 

1249.71 

226.16 

1.407 

.256 

B 

1 194.65 

271.70 

C 

1388.63 

435.64 

There  was  no  significant  difference  among  the  three  designs  on  the  number  of  voluntary  uses  of  the  back  button. 
Table  3.  ANOVA  for  the  number  of  voluntary  back  button  clicks 


Design 

Mean 

SD 

F 

Sig. 

A 

18.92 

14.31 

2.997 

.062 

B 

7.64 

6.89 

C 

16.86 

15.69 

Note  that  the  significance  level  (.062)  approaches  the  .05  level,  and  it  is  possible  that  Design  B (persistent  top-level  menu  along  the  left  side) 
results  in  fewer  uses  of  the  BACK  button,  compared  to  the  other  navigation  models. 

There  was  no  significant  difference  among  groups  with  respect  to  the  appeal  of  the  navigation  model. 

Table  4.  ANOVA  for  subjective  perception  of  appeal 


Design 

Mean 

SD 

F 

Sig. 

A 

12.43 

2.56 

.369 

.694 

B 

13.07 

3.71 

C 

13.53 

3.93 

There  was  no  significant  difference  among  groups  with  respect  to  the  perceived  usability  of  the  navigation  model. 
Table  5.  ANOVA  for  subjective  perception  of  usability 


Design 

Mean 

SD 

F 

Sig. 

A 

9.21 

1.89 

.163 

.832 

B 

9.20 

3.08 

C 

9.73 

3.45 

There  was  no  significant  difference  among  groups  with  respect  to  the  perception  of  Agetting  lost@  in  the  navigation 
model. 


Table  6.  ANOVA  for  subjective  perception  of  Agetting  lost@ 


Design 

Mean 

SD 

F 

Sig. 

A 

13.36 

1.39 

.841 

.832 

B 

13.53 

2.72 

C 

13.93 

3.33 
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We  should  emphasize  that  the  web  site  for  the  School  of  Education  (which  is  similar  to  Design  A in  our 
study,  with  its  bottom-of-page  navigation  system  removed)  was  previously  developed  and  revised  through  extensive 
usability  testing,  making  it  as  easy  as  possible  for  users  to  find  answers  to  typical  information  seeking  tasks  such  as 
those  in  our  study.  The  persistent  or  semi-persistent  menus,  which  were  added  in  Designs  B and  C in  our  study, 
apparently  did  not  significantly  enhance  its  perceived  usability  or  appeal,  which  was  very  good  from  the  start.  With 
web  sites  where  little  or  no  usability  testing  is  used  in  the  design  process,  we  can  speculate  that  persistent  or  semi- 
persistent  menu  systems  might  make  a bigger  difference. 

We  strongly  believe  that  more  research  is  needed  before  we  can  be  sure  that  no  differences  exist  among 
these  different  navigational  models,  because  several  usability  factors  became  apparent  during  the  study.  Factors  that 
pertained  to  a majority  of  respondents  are  discussed  below,  and  each  of  these  behaviors  was  either  observed  by  the 
research  team  or  specifically  mentioned  by  the  subjects. 

First,  most  subjects  were  not  initially  cognizant  of  the  menu  schemes  in  Designs  B and  C.  In  Design  5,  after 
six  to  seven  questions  subjects  typically  started  to  use  the  sidebar  menu  scheme  that  was  provided  in  the  left  column. 
Most  subjects  mentioned  that  they  did  not  notice  the  sidebar  menu  function  until  after  the  tasks  had  started.  Also,  it 
took  answering  a few  questions  before  the  subjects  understood  how  the  side  bar  in  Design  B was  applicable  to  their 
task-searching  behavior.  In  Design  C,  several  subjects  mentioned  that  they  did  not  notice  the  hierarchical  menu  at 
the  top  until  they  had  navigated  the  structure  for  several  levels.  It  took  considerably  more  time  for  subjects  in 
Design  C to  notice  and  begin  to  use  the  menu  chain  at  the  top  of  the  page  versus  the  time  required  for  subjects  in 
Design  B with  the  sidebar  at  the  left  of  the  screen.  One  exception  to  the  time  factor  in  Design  B was  a subject  who 
was  a media  specialist  who  immediately  understood  and  consistently  used  the  menu. 

A majority  of  subjects  mentioned  that  the  exemplars  were  the  most  beneficial  aspect  of  navigation.  Design 
A contained  exemplars  for  all  its  menus.  Designs  B and  C contained  Design  A with  the  addition  of  menu  choices  on 
the  side  or  top,  respectively,  which  did  not  contain  exemplars  due  to  limitations  in  screen  real  estate.  We  can 
speculate  that  if  users  visit  a site  frequently,  they  would  learn  the  meanings  of  the  labels  used  for  the  menu  items  and 
be  less  dependent  on  exemplars  for  deciding  which  part  of  the  hierarchy  they  need  to  go  to.  Our  study  was  relatively 
short  (less  than  an  hour),  and  subjects  were  screened  to  ensure  that  they  were  not  familiar  with  its  structure. 

In  addition  to  the  exemplar  confound,  most  subjects  used  the  home  button  for  navigation  at  the  beginning  of 
each  task.  As  part  of  each  treatment,  the  home  button  was  set  to  go  to  the  home  page  for  that  design.  For  most 
subjects,  it  was  much  easier  to  navigate  to  the  home  page  using  the  home  button  and  then  to  use  the  exemplars  to 
find  specific  pieces  of  information.  This  would  be  atypical  for  most  web  sites,  and  hence  may  have  confounded  our 
results. 

Most  subjects  mentioned  that  during  Design  B the  sidebar  menu  should  have  been  contained  on  the  home 
page  as  well  as  the  other  pages  in  the  document.  Subjects  stated  that  having  the  sidebar  menu  on  sub-levels  and  not 
on  the  main  level  was  confusing.  They  did  not  understand  how  the  side  bar  menu  was  helpful  until  they  discovered 
it,  typically  after  six  to  seven  questions. 

Subjects  were  confused  by  the  dead  links.  In  order  to  keep  subjects  from  leaving  the  experimental  web 
designs  used  in  the  study,  links  to  web  pages  outside  the  navigation  models  were  disabled  intentionally  to  prevent 
confounding  by  other  inconsistent  navigation  schemes  used  on  external  pages.  It  was  not  practical  to  reproduce  the 
entire  School  of  Education  web  site  for  each  of  the  three  treatment  conditions,  which  would  have  consisted  of 
literally  thousands  of  web  pages  developed  by  different  departments  and  units.  Consequently,  several  subjects  gave 
answers  to  the  task  they  were  completing  based  on  the  link  name  with  exemplars  itself  instead  of  clicking  and 
following  the  link. 

Conclusion 

In  summary,  the  results  were  somewhat  disappointing  in  that  we  expected  to  find  designs  B and  C to  be 
more  efficient  with  respect  to  time  to  find  answers  to  information  seeking  tasks.  If  choices  are  visually  present  to 
navigate  by  means  of  shortcuts  to  other  parts  of  a menu  hierarchy,  then  this  increases  the  compactness  of  the 
hypermedia  system  (i.e.,  shorter  distances  between  nodes).  It  also  increases  the  flexibility,  since  users  have  multiple 
ways  to  move  around  a site.  However,  our  data  from  this  study  do  not  support  these  expectations  (with  the  exception 
that  Design  B may  decrease  reliance  on  the  use  of  the  back  button  for  navigation,  since  the  differences  approached 
statistical  significance).  One  could  conclude  that  there  are  indeed  no  differences  among  the  three  types  of  navigation 
models,  and  hence  could  use  other  design  criteria  for  making  a choice  among  them.  For  example,  if  screen  space  is 
needed  to  minimize  scrolling,  then  traditional  non-persistent  menu  structures  are  more  efficient  with  respect  to  use  of 
screen  real  estate.  The  purpose  of  the  web  site  might  also  dictate  which  model  is  preferable.  For  example,  Yahoo  is 
a large  clearinghouse  for  links  to  the  web  (a  Ayellow  pages@  function).  For  such  a large,  deep  hierarchy  of  choices, 
the  use  of  semi-persistent  hierarchical  chains  (or  textual  maps,  as  represented  in  our  Design  C)  occupies  less  screen 
real  estate  than  a sidebar  menu  and  yet  provides  flexibility  for  moving  around  the  hierarchy. 
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We  suggest  that  future  research  should  be  done,  given  constraints  we  faced  in  this  study  and  the  possibility 
of  confounded  results  due  to  those  constraints. 
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Abstract 

Multidimensional  knowledge  structures , described  from  a constructivist  perspective  and  aligned  with  the 
Mind  as  Rhizome  metaphor,  provide  support  for  constructivist  learning  strategies.  In  this  qualitative  study, 
cues  and  trajectories  in  the  children 's  re -presentations  of  knowledge  were  identified.  Implications  of  these 
structures  for  generating  and  developing  multiple  perspectives  and  developing  learner  ownership  are 
discussed. 

Background 

Hypothetical  knowledge  structures  have  been  proposed  and  are  used  to  discuss  and  describe  mental 
processes  from  a cognitive  theoretical  perspective.  Schemata  (Rumelhart  & Ortony,  1977),  frames  (Minsky, 
1974/1997),  semantic  networks  (Woods,  1975),  and  scripts  (Schank,  1975)  provide  a framework  for  describing  how 
knowledge  is  organized  in  memory,  allowing  for  easy  retrieval  of  information  and  playing  a role  in  perception  and 
comprehension.  Schema,  as  an  example  these  knowledge  structures,  are  described  as  existing  for  generalized 
concepts  (Rumelhart  & Ortony,  1977).  Yet,  when  these  structures  are  studied,  they  are  often  isolated  within  a 
domain  of  interest  for  the  researcher,  and,  in  fact,  are  described  as  being  domain  organized  (Bruning,  Schraw,  & 
Ronning,  1995).  For  example,  schema  are  described  for  social  studies  (Duis,  1996;  Tomey-Purta,  1991)  and 
mathematics  (Jitendra  & Hoff,  1996;  Zeman,  1991).  The  role  of  gender  schema  has  been  considered  in  reading  and 
story  memory  (Day,  1994;  Welch-Ross  & Schmidt,  1996). 

Although  a domain  focus  provides  for  description  of  knowledge  and  knowledge  construction  processes 
within  particular  domains,  knowledge  and  learning  that  aligns  with  the  constructivist  perspective  requires  a broader 
look  at  knowledge  structures.  Although  constructing  relationships  related  to  domain  knowledge  is  an  important 
activity  in  the  learning  process,  just  as  important  is  the  learner’s  prior  knowledge  not  related  to  that  domain,  which  is 
somehow  activated  by  information  in  that  domain.  New  information  is  interpreted  by  an  individual  in  terms  of 
existing  knowledge  that  is  divergent  and  unique  to  that  individual.  This  pilot  study  of  elementary  students  begins  to 
look  at  how  knowledge  structures  extend  beyond  the  domain,  thus  providing  for  the  multidimensionality  of 
knowledge  structures. 

Theoretical  Perspective 

The  traditional  cognitivist  perspective,  as  was  initially  aligned  with  the  information-processing  view,  tends 
to  view  knowledge  construction  as  a process  whereby  learners  develop  representations  of  an  ontologically  objective 
reality  (Lakoff,  1987).  An  orientation  to  teaching  that  would  align  with  this  perspective  is  a knowledge  or 
information  transmission  model  (Kember  & Gow,  1994).  Critical  in  the  learning  process  is  the  way  in  which  the 
information  is  organized,  allowing  for  easy  acquisition  and  retrieval.  In  the  instructional  process,  the  information  is 
often  presented  in  final  form  (e.g.,  in  Ausubel’s  (1961)  meaningful  reception  learning),  and  the  learner  is  to 
internalize  this  information  (Driscoll,  1994).  Although  a learner’s  prior  knowledge  and  experience  provide  ways  for 
an  individual  to  link  to  the  new  information,  the  instructional  process  proceeds  from  the  instructor’s  point  of  view 
(Wagner  & McCombs,  1995). 

From  the  constructivist  view,  not  only  is  knowledge  constructed,  as  could  also  be  described  from  the 
cognitivist  perspective,  but  a learner  has  no  alternative  but  to  construct  knowledge  based  upon  his  or  her  own 
experience  (von  Glaserfeld,  1995).  The  constructivist  perspective  is  based  on  a subjectivist  or  interpretivist  view  of 
the  world.  Although  a real  world  is  posited  to  exist,  what  an  individual  knows  of  that  world  is  based  upon  his  or  her 
own  experiences  and  interpretations  of  that  world.  Two  primary  constructivist  views  currently  exist:  cognitive- 
constructivist  where  learning  is  a process  of  active  cognitive  reorganization  and  socio-constructivist  where  learning 
is  acculturation  into  an  established  community  of  practice  (Cobb,  1994;  Duffy  & Cunningham,  1996).  The  current 
study  focused  on  knowledge  structures  of  individual  students,  and  is  more  appropriately  aligned  with  the  cognitive- 
constructivist  view.  From  this  view,  learners  are  to  reshape  or  transform  information,  resulting  in  new  cognitive 
structures  (Brooks  & Brooks,  1993).  Theses  structures,  rather  than  being  prescribed  and  defined  by  a teacher,  are 
idiosyncratic  to  learners  based  upon  their  own  experience  and  prior  knowledge. 

In  constructivist  learning,  the  instructor  acts  as  a facilitator  or  coach,  supporting  students  in  the  learning 
process  by  providing  resources  and  modeling  thinking  skills.  More  critically  in  this  bi-directional  process,  the 
instructor  should  come  to  understand  the  learner’s  point  of  view,  then  pushing  the  learner  to  the  edge  of  his  or  her 
thinking  to  foster  a perturbation  that  will  extend  the  learner  to  expanded  understandings  (Duffy  & Cunningham, 

1996).  Principles  of  constructivist  learning  environments  include  that  of  learner  ownership  where  problems  have 
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relevance  to  the  students,  generation  and  evaluation  of  multiple  perspectives,  and  problem  and  environment 
complexity  appropriate  for  the  learner  and  faithful  to  the  task  (Duffy  & Cunningham,  1996;  Honebein,  Duffy,  & 
Fishman,  1993;  Savery  & Duffy,  1996). 

These  characteristics  provide  a rationale  for  broadening  the  view  of  knowledge  structures,  not  only  beyond 
domains,  but  also  to  capture  the  idiosyncratic  experiences  and  knowledge  of  learners.  Knowledge  is  often  described 
as  being  idiosyncratic  (Alexander,  Schallert,  & Hare,  1991)  thus,  there  needs  to  be  an  opportunity  for  learners  to 
create,  consider,  and  express  their  own  unique  knowledge  as  well  as  descriptions  of  knowledge  structures  that 
capture  the  idiosyncratic  nature  of  knowledge. 

Accessing  Knowledge  Structures 

Typically,  concept  maps  (Beyerbach,  1988;  Shavelson  & Stanton,  1975)  and  semantic  ordered  trees 
(Strahan,  1989)  have  been  used  as  methods  for  gathering  information  about  knowledge  structures.  There  are  variety 
of  ways  in  which  these  have  been  implemented.  For  example,  when  using  structured  concept  maps,  concepts  are 
provided  and  the  learner  is  asked  to  identify  the  links  among  the  concepts.  In  unstructured  maps,  the  learner  is 
provided  the  topic  area  rather  than  individual  concepts.  This  method  can  provide  for  idiosyncratic  views  among 
individuals  (Winitzky,  Kauchak,  & Kelly,  1994).  However,  even  within  the  unstructured  maps  there  may  be  implied 
constraints  to  remain  within  the  topic  or  domain  and  not  identify  meaningful  connections  beyond.  In  fact,  few 
studies  have  avoided  limiting  a learners  knowledge  structures  to  predefined  concepts  (Lederman,  Gess-Newsome,  & 
Latz,  1994).  Ginsburg  (1997)  challenges  that  if  one  is  to  consider  knowledge  from  a constructivist  perspective,  one 
has  no  alternative  but  to  consider  more  non-standardized  forms  of  data  collection,  ones  that  allow  for  the  point  of 
view  of  the  learner. 

Alternative  Perspective 

The  Mind  as  Rhizome  metaphor  provides  an  alternative  perspective  from  which  to  view  knowledge.  A 
biological  example  of  a rhizome  is  a root  crop  such  as  tulips  or  crab  grass,  a system  of  stems,  roots,  fibers  with  fruits 
of  tubers,  bulbs,  and  leaves  (Duffy  & Cunningham,  1996).  When  applied  as  a metaphor  of  mind,  the  rhizome 
provides  a number  of  characteristics  that  aid  understanding  of  mind  from  a constructivist  perspective  and  captures 
the  idiosyncratic  nature  of  knowledge  as  well.  A rhizome  continuously  grows  in  dimension,  changing  its  own  nature 
as  it  increases  in  dimension.  Thus,  the  rhizome  is  a dynamic  system,  one  that  changes  through  time.  Features  within 
the  rhizome  are  heterogeneous,  not  needing  to  be  similar  to  one  another,  each  point  having  the  potential  to  be 
connected  to  any  other  point.  This  potential  connectedness  has  no  hierarchy  or  genealogy,  there  being  no 
subordinate  points  within  the  structure.  A rhizome  has  an  infinite  number  of  entrances,  or  points  from  which  it  is 
viewed,  yet  it  has  not  outside.  Without  an  outside,  it  can  only  be  described  from  the  inside,  from  one  of  the  infinite 
vantage  points  within.  It  is  an  open  network  in  a which  everything  could  be  connected  to  everything  else  within  a 
multidimensional  space  (Deleuze  & Guattari,  1983;  Duffy  & Cunningham,  1996).  Because  of  the  potential  for 
infinite  connections  and  its  dynamic  existence,  each  view  or  perspective  from  within  must  be  idiosyncratic. 

Traversing  the  rhizome,  moving  through  the  maze-like  structure,  is  described  as  a map  (Deleuze  & Guattari, 
1983),  although  a path  may  provide  a better  way  to  conceptual  this  process.  A path  through  the  rhizome  (a  trajectory 
through  the  dynamic  system)  is  not  static,  but  a continuous  construction  capturing  knowledge  as  a process.  In 
contrast,  if  one  considers  that  a concept  map  could  capture  a portion  of  the  rhizome,  knowledge  appears  a static 
commodity.  A concept  map  seems  to  sever  the  connections  that  exist  between  the  concept  map  and  the  rhizome 
from  which  it  was  removed.  Thus,  knowledge’s  idiosyncratic  nature  is  suppressed.  Knowledge,  when  considered  as 
a dynamic  path  or  trajectory  through  the  rhizome,  is  affected  not  only  by  new  information  it  encounters,  but  the 
entire  path  from  which  it  emerged.  The  multidimensional  rhizome  of  infinite  connections  captures  a mind  that 
includes  past  experience  and  knowledge  as  a basis  for  a continuing  construction  process  as  new  information  and 
experience  are  encountered. 

When  considering  the  multidimensionality  of  the  rhizome,  it  is  not  difficult  to  speculate  what  the 
dimensions  of  multidimensional  knowledge  could  be.  At  a minimum,  time  is  an  added  dimension.  Alexander  and 
Murphy’s  (1998)  description  of  an  individuals  knowledge  base,  based  on  learner-centered  psychological  principles 
aligned  with  the  constructivist  perspective,  included  sociocultural,  strategic  abilities,  personal  beliefs,  and  goals  as 
interactive  dimensions.  Ginsburg  (1997)  stated  that  “cognition  without  emotion  is  vacuous”  (p.  65),  adding  another 
dimension.  These,  and  likely  many  others,  provide  a trajectory  or  path,  reflecting  the  many  types  of  connections  an 
individual  can  construct  at  any  given  time,  based  on  personal  experience  and  knowledge.  Any  one  of  these 
dimensions  may  be  encountered  and  followed  at  any  time  during  learning.  When  studying  knowledge  structures,  all 
types  of  trajectories  should  be  considered  relevant.  Domain  or  subject  areas,  along  with  these  other  dimensions,  do 
provide  potential  for  infinite  connections  with  unique  past  experience  and  knowledge  of  learners.  The  rhizome 
metaphor  with  its  heterogeneous  infinite  connections,  coupled  with  data  gathered  through  non-traditional  methods  as 
suggested  by  Ginsburg,  provides  a framework  for  study  of  a more  divergent  look  at  knowledge. 
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Purpose  and  Guiding  Question  for  Study 

The  purpose  of  this  study  was  to  begin  description  of  hypothetical  knowledge  structures  based  on  empirical 
data  gathered  in  learning  situations.  The  rhizome  metaphor  provided  a framework  from  which  characteristics  of 
broader  knowledge  structures  could  be  described.  The  study  sought  a description  of  knowledge  structures  that  (a) 
were  distributed,  allowing  for  heterogeneous,  multiple  connections  that  existed  within  one’s  knowledge,  and  (b) 
contained  connections  that  extended  beyond  domain-specific  boundaries  thus  adding  dimensions.  The  broad  guiding 
question  for  this  study  was:  How  can  knowledge  of  these  elementary  school  children  who  were  situated  in  an 
existing  school  environment  be  described  as  multidimensional  knowledge  structures? 

Methodology 

A qualitative  study  was  conducted  seeking  empirical  support  for  a description  of  multidimensional 
knowledge  structures,  focusing  on  the  idiosyncratic  nature  of  knowledge  that  arose  in  the  classroom  and  was  based 
upon  children’s  individual  prior  experience  and  knowledge. 

Participants 

Participants  for  this  study  were  a first  and  a sixth  grade  parochial  classroom  in  a small  Midwestern  city. 

The  first  grade  class  had  14  boys  and  1 1 girls,  ages  6 to  7,  of  which  13  consented  to  participate  in  the  study  (8  boys 
and  5 girls).  The  sixth  grade  class  had  9 boys  and  15  girls,  ages  12  and  13,  with  3 boys  and  9 girls  participating  in 
the  study.  Prior  to  the  beginning  of  data  collection,  the  researcher  visited  the  classrooms,  was  introduced  to  the 
students,  and,  in  the  first  grade  class,  interacted  with  the  children  by  assisting  them  with  their  class  work.  The  first 
grade  teacher  stated  that  these  early  encounters  with  the  children  identified  the  researcher  as  a helper  and  increased 
the  children’s  comfort  level  and  willingness  to  talk  in  the  interviews.  The  sixth  grade  teacher  was  used  as  a 
secondary  participant. 

Procedure 

Data  was  gathered  through  a variety  of  sources  using  a number  of  methods  as  appropriate  for  a qualitative 
study  (Lincoln  & Guba,  1985).  Each  classroom  was  observed  three  times,  the  first  grade  during  a variety  of  subjects 
and  the  sixth  grade  during  social  studies.  In  the  initial  design  for  the  study,  instances  of  idiosyncratic  comments  and 
questions  by  students  were  sought.  Students  were  to  be  interviewed  about  the  source  of  these  idiosyncratic 
comments  and  questions,  seeking  connections  between  them  and  the  new  information.  After  the  first  observation  in 
each  class  it  became  clear  that  these  instances  would  not  occur  in  any  abundance,  perhaps  resulting  from  the  didactic 
nature  of  the  classrooms.  The  design  evolved  so  that  interview  students  were  randomly  chosen.  Nine  first  grade 
students  and  three  sixth  grade  students  were  interviewed.  The  individual  interviews  were  conducted  after  the 
classroom  observation,  were  audio  taped,  and  later  transcribed. 

Each  semi-structured  interview  began  by  having  the  child  describe  what  he  or  she  remembered  from  the 
class  session.  Open-ended  questions  followed,  asking  the  child  about  idiosyncratic  comments  that  he  or  she  may 

have  brought  up  during  class  (e.g..  During  your  lesson  today  you  brought  up . What  made  you  think  of  that?) 

or  may  have  had  covertly  (e.g.,  At  times,  when  we’re  learning  one  thing,  other  things  pop  into  our  head  that  are  not 
what  about  we’re  learning.  Sometimes  we  hear  the  teacher  say  might  remind  us  of  something  else  we  know.  While 

you  were  studying this  morning,  did  anything  like  that  pop  into  your  head?).  The  interview  followed  from  the 

child’s  comments,  each  interview  lasting  less  than  15  minutes. 

The  sixth  grade  students  also  completed  a brief  writing  activity  on  the  Vikings,  their  topic  that  day  in  social 
studies.  A 15  minute  introduction  to  the  writing  task  allowed  the  students  to  ask  questions  about  what  they  were  to 
do  and  helped  ensure  their  understanding.  Students  were  provided  open-ended  prompting  questions  to  help  them 
begin  the  activity  (e.g..  Describe  the  conquests  of  the  Vikings.)  and  were  told  that  if  something  popped  into  their 
heads,  regardless  of  its  relationship  to  social  studies,  to  follow  that  thought  and  write  about  the  new  topic.  Students 
were  asked  to  include  why  the  new  topic  was  related  and  were  provided  a number  of  example  transition  sentences  to 

aid  this  process  (e.g.,  This  reminds  me  of because .).  Writing  time  was  extended  from  the  30  minutes 

initially  planned  to  45  minutes  as  the  students  continued  working  on  the  activity.  Students  were  able  to  use  their 
textbooks  for  the  writing  activity  to  help  them  with  ideas  and  spelling  and  were  asked  to  indicate  if  they  used  their 
book  none,  some,  or  a lot. 

The  sixth  grade  teacher  was  interviewed  following  student  data  collection.  This  interview  was  a means  to 
further  triangulate  the  data  and  gain  further  understanding  if  some  of  the  links  that  the  students  provided  in  their 
writing  and  interviews  had  occurred  previously  in  classroom  instruction  or  in  school.  In  this,  the  role  of  the  teacher 
was  not  to  confirm  or  deny  the  students’  knowledge  construction,  but  to  provide  information  about  the  basis  of  some 
of  the  connections. 
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Analysis 

Analysis  proceeded  as  typically  done  in  a qualitative  study,  seeking  trends  among  the  data  from  the  various 
sources  (Bogdan  & Biklen,  1982;  Lincoln  & Guba,  1985).  Data  from  the  observations,  interviews,  and  writing 
activities  were  reduced  to  series  of  propositions  (Novak  & Musonda,  1991),  generally  single  sentences  or  topic 
areas.  These  were  then  reviewed  for  points  or  ideas  from  which  one  topic  or  idea  moved  to  the  next. 

Results 

Heterogeneous , Multiple  Connections:  Cues 

Two  primary  trends  emerged  through  the  analysis.  First,  within  the  children’s  writing  and  interview  data 
there  existed  cues,  stimuli  that  allowed  the  child  to  turn  a mental  or  verbal  comer,  digress  from  the  topic  at  hand  and 
follow  a path  to  a related  concept  or  idea.  Although  these  cues  seemed  similar  to  the  cues  described  from  the 
cognitive  learning  perspective  used  to  aid  information  storage  and  retrieved  (Bruning  et  al.,  1995)  they  were  not 
teacher  defined  or  mediated  (i.e.,  a part  of  the  pre-planned  structure  of  the  topic  to  be  learned).  These  cues  occurred 
either  in  the  learning  environment,  generally  from  a teachers  comment  or  question,  or  from  a child’s  own  thoughts 
(verbally  or  in  writing)  as  a type  of  re-presentation  (in  the  Piagetian  sense)  where  individuals  replay  or  reconstruct 
their  knowledge  to  themselves  (von  Glaserfeld,  1995).  For  this  study,  cues  often  appeared  as  single  words  or 
concepts,  making  it  clear  where  the  learner  turned  a cognitive  comer  or  chose  a path  in  the  rhizome  that  was  different 
from  what  the  teacher  was  presenting  or  what  the  learner  was  currently  writing.  Analysis  identified  the  following 
cues:  sounds  like  (two  words  that  sounded  alike  but  shared  no  meaning),  looks  like  (a  visual  cue  that  looked  like 
something  else),  same  word  but  different  concept  (a  word  or  derivation  of  a word  that  described  two  different  ideas), 
same  concept  but  different  context  (an  idea  or  concept  that  occurred  or  existed  in  other  circumstances),  series  (an 
itemized  set  of  words),  and  complex  relationships  (an  inference  is  an  example). 

An  observation  of  Tim’ si  first  grade  reading  group  provides  an  example  of  a same  word  but  different 
concept  cue.  Tim  has  a different  conception  of  the  word  “meadow”  when  it  is  presented  as  a new  word. 

The  teacher,  introducing  the  story  The  Frog  and  the  Toad,  asked  the  small  first  grade  reading  group  what  a meadow  was. 
Meadow  was  a new  word  in  the  story. 

“A  meadow  is  a place  with  a few  buildings,”  Tim  said. 

The  teacher  paused,  “Meadow  Wood,  where  your  grandpa  lives,”  she  interpreted  his  response. 

“Animals  live  in  a meadow,”  another  student  offered. 

“There’s  a fountain  in  a meadow,”  Tim  said. 

The  teacher  tried  to  clarify,  “They  call  it  Meadow  Wood  because  it  looks  like  a meadow.” 

The  students  offered  other  descriptors  of  meadow.  “A  meadow  is  flat.”  “A  meadow  has  flowers.”  “A  meadow  has 
deer.”  “A  meadow  has  raccoons.” 

The  teacher  added,  “A  meadow  has  possum.”  Still  addressing  Tim’s  idea  about  meadows,  she  said,  “In  Meadow  Wood 
sometimes  animals  come  near  the  buildings  and  eat  the  plants.  A meadow  is  a flat,  grassy  place.”  (observation,  grade  1 , 

April  6,  1998) 

It  is  clear  that  Tim’s  definition  of  the  word  meadow,  although  the  same  word,  does  not  match  that  of  the 
teacher’s.  Although  Tim’s  idiosyncratic  definition  of  the  word  is  incorrect  based  on  the  teacher’s  definition,  it  is  a 
definition  consistent  with  his  prior  experience  and  knowledge.  As  the  teacher  tries  to  clarify  this,  it  becomes  clearer 
that  although  the  same  word,  Tim  has  a different  concept  in  mind. 

The  interviews  and  writings  of  the  sixth  grade  students  provided  examples  of  more  sophisticated  cues.  Beth 
brings  up  a question  in  her  interview  that  she  wondered  about  during  class,  “If  we  had  a test  question  on  what 
happened  on  Christmas  day  in  the  800s  would  it  be  both  answers?”  (Beth,  age  12,  April  13,  1998).  Her  question 
occurred  in  a lesson  on  Charlemagne  where  the  students  were  told  the  Charlemagne  had  been  crowned  Holy  Roman 
Emperor  on  Christmas  day.  Broader  than  being  the  same  word,  as  was  Tim’s  case,  Christmas  is  a concept,  or  idea. 
Beth  had  the  same  concept  of  Christmas  as  the  teacher.  Her  question  indicated  that  this  concept  can  extend  beyond 
contexts  (that  of  learning  a particular  event  in  social  studies  in  the  classroom  and  that  the  traditional  view  of 
Christmas). 

In  the  same  sixth  grade  lesson,  Anne’s  interview  provided  another  concept  that  was  touched  on  in  class  that 
she  moved  into  different  contexts. 

I asked  Anne,  “When  you’re  sitting  in  class,  or  when  you’re  sitting  anywhere,  but  we’ll  kind  of  focus  on  social  studies 
today  (yeah)  and  you’re  hearing  things,  like  Mrs.  Larson  is  asking  questions,  and  you’re  discussing  the  topic,  or  you’re 
looking  at  the  book  or  whatever,  a lot  of  times  other  things  will  pop  into  our  minds  (uh  huh)  so  I was  wondering  if 
during  social  studies  anything  popped  into  your  mind.” 

“Well,  I have  a track  meet  this  afternoon  so  I was  thinking  about  if  it  was  going  to  be  rained  out  or  not.  (“OK,”  I 
inteijected.)  and  I was  thinking  about  how  I went  to  the  art  museum  in  Chicago  when  she  mentioned  art.”  (Anne,  age  12, 

April  13, 1998) 
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1 Student  and  teacher  names  are  pseudonyms. 


The  teacher  had  mentioned  that  art  was  encouraged  in  the  schools  during  Charlemagne’s  time  and  that 
churches  were  like  museums  because  of  the  artwork  in  them.  Since  the  mention  of  art  in  the  class  was  at  the  end  of 
the  lesson,  perhaps  Anne’s  recall  could  be  attributed  to  a recency  effect.  But,  later  in  the  interview,  when  prompted 
about  the  most  interesting  thing  about  the  class,  she  returned  to  the  topic  of  art  showing  that  she  had  considered  the 
concept  a bit  beyond  the  mention  in  class,  making  a comparison  to  art  work  in  the  churches  now,  although  not 
expanding  any  further. 

“I  thought  the  art  work  in  the  church  was  because  usually  these  days  all  you  see  is  stained  glass  and  everything 
(OK)  and  you  don’t  really  see  anything  except  like  the  stations,  but  you  never  really  see  any  art  work.” 

“OK,  why  do  you  suppose  that  is?” 

“I  don’t  know.”  (Anne,  age  12,  April  13,  1998) 


Anne  mentioned  twice  in  the  interview  that  social  studies  was  boring.  When  called  on  to  answer  a question 
during  class,  Anne  had  to  ask  the  teacher  to  repeat  the  question,  then  thumbed  through  her  book,  and  eventually 
provided  an  answer.  Another  sixth  grade  student,  who  said  that  history  was  her  favorite  subject,  identified  a different 
concept  that  she  extended  beyond  the  current  context  in  her  re-presentations.  Wendy  focused  on  the  wars  that 
Charlemagne  had  fought,  being  able  to  identify  his  foes. 

Well,  in  history  I was  thinking,  because  I really  looked  over  the  chapter  really  good.  1 was  really  thinking  more 
about  all  the  wars  we’ve  been  through  with  other  countries,  Vietnam  War  and  World  War  1 and  World  War  n,  and  I 
started  thinking  about  the  Titanic  ship,  because  I read  a lot  on  that,  and  how  there  were  three  boats  named  Titanic 
and  they  all  sank.  So,  I was  thinking  if  that  was  a curse  name  or  something.  And  then  she  started  talking  about 
how  they  switched  the  Franks  and  stuff  and  I thought  about  that  in  history,  and  1 eventually  when  back  to  thinking 
about  other  stuff.  (Wendy,  age  12,  April  13,  1998) 

When  asked  later  in  the  interview  to  talk  more  about  the  idea  of  wars  that  were  triggered,  asking  if  she  saw 
relationships  between  the  two,  Wendy  continued  following  a path  among  her  areas  of  interest.  The  concept  of  wars 
is  extended  to  fighting  and  the  use  of  peace  treaties,  again  returning  to  a war  that  she  had  mentioned  earlier. 

Yeah,  I think  we  still  Fight,  except  we  fight  a lot  more  about  stupider  things,  like  religion,  1 know  it’s  important  but 
it’s  not  something  we  should  really  Fight  over,  we  should  just  get  along  and  try  to  like  divide  it.  1 watch  a lot  of 
videos  because  my  mom  works  for  National  Geographic  company  and  if  she  sells  so  many  products  she  gets  a 
video.  So  I sit  down  and  watch  it.  Like  the  Titanic  one.  That  was  a little  boring  because  it  was  mostly  stuff  1 
knew.  But  when  it  was  about  Egypt  and  stuff  and  talked  about  the  mummies  1 thought  that  was  cool.  1 think  they 
considered  things  more  about  violence  because  you’re  basically  surrounded  by  it.  Basically,  like  guns  and  all  that, 
like  kids  killing  other  kids  and  the  wars  going  on  around  us.  I know  they’re  trying  to  have  a peace  treaty  with,  1 
foiget  with  what  country,  and  I don’t  think  peace  treaties  work  because  they  break  them  like  when  Hitler  broke  the 
one  in  WW  n.  They  usually  keep  them  for  a couple  of  years,  or  a decade  or  so  and  then  they’ll  break  them  which  is 
not  veiy  good.  I think  if  you  make  a promise  you  should  live  up  to  your  promise  unless  something  really  big  comes 
along.  (Wendy,  age  12,  April  13,  1998) 

Wendy  later  continued  by  talking  about  freedom,  linking  the  idea  of  being  forced  to  change  religion  (as  was 
done  with  Charlemagne’s  conquests)  and  the  idea  of  being  able  to  choose  things  for  herself,  such  as  books  to  read. 
Although  the  tangents  of  discussing  Egypt  and  the  Titanic  in  the  interview  seem  irrelevant,  two  weeks  later  during 
the  sixth  grade  writing  activity  on  the  Vikings  (where  Wendy  stated  that  she  used  the  book  some)  she  returns  to  the 
same  topics. 


I like  the  Vikings  tons  but  who  could  beat  the  Romans.  Talk  about  cool.  They  rule  everything.  They  had 
everything  right  till  their  good  ruler  died  then  they  had  their  downfall.  Egypt  was  cool  too.  1 liked  how  they 
wrapped  their  dead  up.  My  great  uncle  just  died  it  was  sad.  1 can’t  turn  my  book  to  the  Egypt  pgs  at  school 
because  on  3 pages  it  [was]  all  muddy.  I did  not  mean  to  do  it.  I left  my  book  open  and  my  dog  jump  with  his 
muddy  paws  all  over  it.  I’m  planning  to  tell  Mrs.  Larson  the  last  day  of  school  and  I will  bring  $5  to  pay  for  it.  I 
just  don’t  want  to  be  yelled  at.  I hate  that.  Egypt  is  hot  but  they  have  nice  piramids  and  status  [statues].  The 
Egypt  people  had  grave  takers.  That  would  be  mean  to  take  some  ones  thing  and  not  let  them  have  their  peace.  I 
know  I would  hate  it  if  anyone  took  anything  special  to  me  like  my  bennie  babies,  titanic  stuff,  or  radio.  (Wendy, 
age  12,  April  27,  1998) 

The  linear  format  of  writing  allows  only  one  cue  to  be  followed  at  a time.  But,  Wendy’s  return  to  the  topic 
at  hand,  as  well  as  her  returning  to  a topic  two  weeks  later,  is  a way  that  multiple  connections  to  the  same  concept 
can  be  captured. 

These  few  examples  of  cues,  of  which  76  were  identified  in  the  data,  indicate  how  simple  a cue  can  be.  Yet, 
a cue  has  the  potential  for  multiple  heterogeneous  connections  stemming  from  it.  For  example,  the  concept  of  “art” 
in  Anne’s  thoughts  did  not  have  only  one  connection,  but  many.  Further,  the  types  of  connections,  the  context  where 
they  initially  developed,  were  different.  Beth  was  building  a number  of  connections  stemming  from  the  Christmas 
cue.  Wendy  had  a number  of  connections  with  the  concept  of  war.  That  these  cues  are  not  singular  concepts, 
although  reported  as  such,  suggests  distributedness.  Anne’s  knowledge  about  art  is  not  a self-contained  singular 
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concept  described  in  a hierarchical  structure.  Neither  is  Wendy’s  knowledge  about  war.  When  comparing  Wendy’s 
interview  and  writing,  we  see  that  some  concepts  reoccur  (e.g.,  Egypt),  yet  the  context  and  description  of  them  has 
changed,  based  upon  the  current  reporting  and  situation.  While  Tim  seemed  to  have  only  one  connection  to  the  word 
meadow,  we  can  speculate  that  perhaps  after  the  class  he  had  another.  Tim’s  meadow  cue,  whether  described  as 
correct  or  incorrect,  as  well  as  the  other  cues  identified  that  might  have  been  categorized  as  being  off-track,  reach 
beyond  the  topic  at  hand,  perhaps  even  beyond  the  domain.  The  cues  provide  the  prompt  to  move  beyond  a topic 
and  connect  the  information  in  an  idiosyncratic  way,  a comer  to  a different  dimension  in  ones  knowledge. 

Beyond  Domains:  Trajectories  and  Dimensions 

Once  a cue  was  encountered,  the  learner  followed  a path,  or  trajectory,  through  the  current  slice  of  the 
rhizome.  If  a cue  is  a cognitive  comer,  the  trajectory  is  the  street  that  the  learner  is  on  once  the  comer  is  turned. 
Rather  than  describing  the  trajectory  by  the  actual  script  of  experiences  and  thoughts,  a trajectory  may  be  better 
described  by  the  nature  of  the  path,  or  the  dimension  at  which  the  learner  arrived.  The  nature  of  this  path  could  be 
characterized  in  a number  of  ways,  based  on  the  type  of  experience  and  who  the  players  in  the  experience  were. 
Trajectories  identified  in  the  data  were:  experience  (activity),  personal  (thoughts  and  ideas),  family,  friends,  school, 
society,  emotion/affect,  and  abstract  ideas. 

Following  the  observation  of  the  first  grade  reading  group,  Tim  was  interviewed.  He  confirmed  the 
observation  made  by  the  teacher,  Tim  derived  his  meaning  of  the  word  meadow  from  personal  experience  based  on 
where  his  grandfather  lived. 

Tim  gave  me  a summary  of  The  Frog  and  the  Toad,  the  story  his  reading  group  had  discussed,  which  had  contained 
a new  reading  word:  meadow.  I asked  “I  remember  you  were  talking  about  your  grandpa.  What  made  you  think 
about  that?” 

“I  don’t  know.  When  she  said  ‘meadow’  I thought  of  that  one.” 

‘‘Yeah,  because  he  lives  in  place  that’s  called  Meadow,”  I paused,  not  remembering. 

‘‘Wood,”  he  quietly  interjected. 

“What  was  it  called,  Meadow  Wood?” 

“For  old  people  that  have  strokes.”  he  continued.  “He  had  a stroke  and  he’s  paralyzed  and,  um,  we  just 
go  there  to  visit  him.” 

“Oh,  sounds  like  a nice  place  for  him.” 

“Yeah,  it  has  a water  fountain,  there’s  a dining  place  where  he  used  to  be  able  to  go  out  and  get  food 
there,  when  my  great-grandma  was  alive,  but  she  died.  And  he  can’t  go  there  any  more  because  his  foot’s  too 
swollen.” 

“Oh.” 

“There’s  lots  of  neat  things  there.” 

“Yeah.  They  called  it  Meadow  Woods  and  your  teacher  said  that’s  maybe  because  it  used  to  be  a 
meadow,  right.” 

“Used  to  be  a meadow,”  he,  again  very  quietly,  said. 

“So  what’s  a meadow  like?”  I asked. 

“I  don’t  know.  It’s  pretty  calm,  quiet.”  (Tim,  age  6,  April  6,  1998) 

Although  it  is  not  clear  if  Tim  ever  developed  the  teachers  definition  for  the  word  meadow,  it  seems  clear 
that  Tim’s  conception  of  meadow,  and  the  context  from  which  it  emerged  was  a particularly  meaningful  connection 
of  Tim’s.  The  meadow  cue  lead  to  a rich  description  characterized  by  a number  of  dimensions.  Tim’s  trajectory,  the 
nature  of  the  path  that  the  cue  “meadow”  lead  him,  could  be  characterized  as  an  experience  because  he  had 
experienced  Meadow  Wood  first  hand  and  visited  his  grandfather,  and  also  has  a family  dimension.  Although  Tim 
doesn’t  express  emotion  overtly  about  this  grandfather,  his  way  of  communicating  and  his  mentioning  sickness  and 
death,  provided  an  emotion/affect  dimension.  Tim’s  trajectory  extended  beyond  the  domain  area  of  reading,  the 
story  of  The  Frog  and  the  Toad,  and  the  new  word  “meadow.” 

Anne’s  trajectory  on  art  could  be  classified  in  multiple  ways  as  well.  It  followed  from  an  experience,  but 
also  included  society  (art  in  churches)  and  also  school  given  that  she  attends  a parochial  school.  Beth’s  Christmas 
cue  likewise  had  both  society  and  school  connections.  Although  she  didn’t  specify,  we  can  also  speculate  that  there 
would  also  be  experience  and  family  connections. 

Wendy’s  re-presentations  provide  an  example  of  ties  both  within  and  beyond  a domain.  The  nature  of  the 
paths  largely  focused  on  society  and  school  (although  the  wars  she  mentioned  had  not  been  studied  in  the  current 
school  term)  as  well  as  personal  (without  an  identified  experience).  Wendy’s  ability  to  synthesize  ideas  is 
instructional ly  encouraging,  indicating  a number  of  possibilities  for  considering  knowledge  in  a multidimensional 
way,  synthesizing  within  the  domain,  as  well  as  reaching  outside  of  it  to  broader  societal  issues.  Although  history 
was  Wendy’s  favorite  subject,  she  struggles  with  it  and  her  other  school  subjects  when  it  comes  to  assessment.  Her 
interest  and  synthesis  abilities  do  not  seem  to  emerge  in  the  classroom. 

Cues  are  easy  to  identify,  often  being  of  singular  type  and  usually  simple  (such  as  a single  word). 
Trajectories  are  more  difficult  to  capture,  given  the  number  of  ways  the  nature  of  the  path  could  be  described.  While 
it  seems  redundant  to  classify  a trajectory  as  both  experience  and  family  for  example,  the  further  classification 
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provides  a more  accurate  description  of  the  context  that  surrounds  the  trajectory.  When  considering  another  context, 
notions  like  Tim’s  view  of  meadow  can  no  longer  be  incorrect,  only  idiosyncratic. 

When  considering  cues  and  trajectories  in  relation  to  the  rhizome  metaphor,  they  begin  to  provide  a 
description  of  a slice  of  the  rhizome,  a local  description  from  the  view  of  the  individual.  A cue  is  that  point  from 
which  there  can  be  an  infinite  number  of  connections.  At  that  point,  the  trajectory  that  will  be  followed  is  not  only 
based  upon  that  cue,  but  the  infinite  number  of  connections  that  exist  with  that  cue.  The  path  a learner  follows  at  the 
moment  relies  on  past  experience  and  knowledge  that  have  been  parts  of  previous  trajectories,  each  leaving  “cue 
residue”  that  can  be  connected  at  any  time  should  the  context  and  new  information  be  remotely  appropriate.  These 
cues  are  not  teacher  defined.  Only  the  meadow  cue  reported  here  might  be  considered  an  intentional  cue,  the  teacher 
trying  to  build  cues  by  building  vocabulary.  Both  classrooms  were  didactic  in  nature  and  the  teachers  were  seeking 
answers  that  had  been  defined  a priori  to  their  questions.  The  trajectories  of  the  students  were  not  a part  of  the 
intended  path  that  the  teacher  would  have  set  through  the  rhizome  yet  provided  potential  relevant  connections  for 
that  particular  child.  This  brings  up  compelling  questions  about  the  implications  for  instruction  from  a constructivist 
perspective,  a view  that  should  find  value  in  the  nature  of  these  self-identified  cues  and  idiosyncratic  trajectories. 

Implications  for  Instruction 

Within  the  constructivist  perspective  a number  of  instructional  strategies  have  been  discussed,  including 
problem  based  learning  (Savery  & Duffy,  1996),  cognitive  apprenticeships  (Collins,  Brown,  & Newman,  1989),  and 
anchored  instruction  ((CTGV),  1990).  Principles  that  these  instructional  strategies  embrace  include  that  of  learner 
ownership,  generating  and  gaining  multiple  perspectives,  providing  a range  of  contexts  in  which  the  concept  might 
appear,  anchoring  to  a larger  task  or  problem,  and  providing  for  reflection  on  the  process  and  content  (Honebein  et 
al.,  1993). 

A number  of  these  characteristics  provide  a framework  for  the  role  of  cues  and  trajectories  in  instruction. 
Learner  ownership,  taking  responsibility  for  his  or  her  own  learning  and  performance,  is  something  which  cannot  be 
directly  handed  to  a learner  (Honebein  et  al.,  1993).  In  fact,  oftentimes,  learners  will  reject  the  specific  learning 
objective  that  might  be  specified  by  the  teacher  (Savery  & Duffy,  1996).  In  this  study,  it  is  difficult  to  imagine  that 
Anne,  for  example,  accepted  and  owned  the  predefined  instructional  goal  in  the  study  of  Charlemagne,  considering 
her  boredom  with  the  class  and  her  inattentiveness.  However,  even  for  Anne,  there  was  a cue,  or  hook,  that  could 
potentially  provide  relevance  for  her  learning.  Art  is  a concept  that  Anne  had  some  interest  in  and  could  be  used  to 
provide  her  an  opportunity  to  find  relevance  in  her  learning  of  a boring  subject.  The  didactic  nature  of  the  classroom 
did  not  allow  for  this  opportunity  to  occur  and,  in  her  summary  of  the  lesson,  Anne  seems  to  know  little  about  the 
topic.  A self-defined  cue,  such  as  art  for  Anne  or  any  of  the  cues  that  occurred  in  Wendy’s  re-presentations,  could 
provide  avenues  whereby  ownership  could  be  developed  by  learners  within  the  instructional  process.  Although  a 
teacher  cannot  know  what  simple  ideas  or  words  might  serve  as  cues  for  each  learner,  the  facilitator  role  allows 
interaction  to  occur  where  a student  may  be  willing  to  express  a cue  and  what  trajectories  might  be  followed — 
providing  personal  relevance  to  the  topic.  The  teacher  can  then  guide  the  learner  to  purposefully  choose  a trajectory 
that  is  relevant  for  the  learner  as  well  as  appropriate  within  the  curriculum. 

We  could  consider  that  Tim  actually  clung  to  the  notion  of  learner  ownership  as  he  kept  trying  to  use  his 
meadow  as  the  meadow  the  teacher  was  talking  about,  offering  his  characteristics  of  meadow  from  his  perspective. 
His  personal  definition,  although  seemingly  incorrect,  provided  an  opportunity  for  him  to  add  personal  relevance  to 
his  learning.  Further,  Tim  provided  a perspective  different  than  the  teachers,  his  own.  Generating  and  gaining 
multiple  perspectives  can  only  come  after  a child  has  developed  his  or  her  own  perspective.  Further,  learners’ 
perspectives  must  be  considered  to  be  viable.  What  is  the  chance  of  generating  multiple  perspectives  if  a perspective 
is  immediately  labeled  by  another  as  not  viable? 

Anne,  Wendy,  and  Beth’s  interviews  were  all  held  after  the  same  class  lesson  on  Charlemagne.  Yet,  each  of 
them  had  different  self-selected  cues  that  provided  them  an  opportunity  to  extend  beyond  the  class  topic  and  the 
explicit  information  presented.  As  each  child  provided  a brief  synopsis  in  her  interview  of  what  had  been  covered  in 
class,  each  remembered  different  aspects,  personally  relevant  because  of  her  own  experiences,  knowledge,  concerns, 
and  interests.  Not  surprisingly,  Beth’s  synopsis  contained  a statement  about  Charlemagne  being  crowned  Holy 
Roman  Emperor,  Wendy’s  included  a listing  of  his  wars  and  who  he  fought,  and  Anne  mentioned  the  artwork  in  the 
church.  Each  could  develop  a different  perspective  on  the  topic  if  provided  the  opportunity.  Sharing  these 
perspectives  could  lead  to  gaining  multiple  perspectives.  But,  for  that  to  occur,  each  child  would  need  the 
opportunity  to  further  expand  their  personal  view  of  the  topic  into  a perspective. 

A methodological  question  arises  when  one  considers  that  the  re-presentations  described  in  the  interview 
did  not  actually  occur  previously  in  the  classroom  as  reported.  Certainly,  it  is  possible  that  the  students  (other  than 
Tim)  could  not  have  had  these  re-presentations  in  class  at  all.  But,  perhaps  it  is  not  important.  The  fact  that  cues  and 
trajectories  could  exist,  whether  in  the  classroom  or  in  a private  interview,  indicates  that  the  connections  between 
personal  experience  and  knowledge  to  the  new  information  are  easily  accomplished.  Perhaps  learners  have  been 
trained  to  self-regulate  their  learning  too  well,  not  offering  personal  links.  What  is  needed  is  support  for  students  to 
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thoughtfully  consider  meaningful  cues  based  on  personal  experience  and  synthesize  the  trajectories  with  the  new 
information.  Rather  than  convergence,  divergence  and  a broader  view  of  knowledge  is  needed  both  for  teachers  and 
learners. 

It  seems  that  learner  ownership  and  generating  and  gaining  multiple  perspectives  will  go  hand  in  hand.  It 
also  seems  that  a likely  place  for  these  to  develop  is  by  providing  students  opportunities  to  self-identify  cues  and  then 
follow  a personal  trajectory.  Despite  a didactic  teaching  environment,  cues  did  emerge  on  their  own.  The 
fruitfulness  of  the  trajectories  that  a learner  may  choose  to  follow  will  only  become  evident  if  the  instructional 
method  provides  an  opportunity  for  it  to  be  followed  and  considered.  Given  the  infinite  number  of  entrances  into  the 
rhizome,  each  one  unique  to  an  individual  learner,  it  seems  like  it  might  be  easier  to  consider  knowledge  structures  as 
multidimensional  than  to  constrain  them. 
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Abstract 

This  study  investigated  the  relationships  between  hypermedia  users'  information  processing  styles  and 
navigational  patterns.  Three  aspects  of  navigational  patterns  were  investigated:  (a)  navigational  depth 
patterns  which  reveal  how  comprehensively  users  access ; (b)  navigational  path  patterns  which  display 
what  sequences  users  follow , and  (c)  navigational  method  patterns  which  show  what  methods  users  employ 
when  using  the  system. 

Information  processing  styles  were  measured  by  the  Human  Information  Processing  Survey.  The  subjects 
were  102  undergraduate  students  enrolled  in  management  courses  at  a university.  Participation  was 
voluntary.  34  students  were  selected  for  each  of  left,  right , and  integrated  information  processor  group. 
The  subjects  interacted  with  A.g.i.l.e.  Resource™  program  to  complete  two  types  of  searching  tasks,  open- 
ended  and  closed-ended  tasks. 

Findings  indicated  that: 

1.  Significant  differences  were  found  among  groups  in  navigational  depth  patterns . The  right  dominant 
information  processing  style  subjects  accessed  significantly  more  new  nodes  than  did  the  left  dominant  information  processing  style 
subjects , 

2.  There  was  a significant  relationship  between  information  processing  style  and  navigational  path  pattern. 
The  right  dominant  information  processing  subjects  followed  significantly  mure  linear  paths  than  the  integrated  information 
processing  style  subjects.  The  right  dominant  information  processing  subjects  also  followed  a lot  more  linear  paths  than  the  left 
information  processing  style  subjects,  although  it  was  not  statistically  significant. 

3.  There  was  a significant  relationship  between  information  processing  styles  and  navigational  method 
patterns.  The  left  dominant  information  processing  style  subjects  employed  significantly  more  analytical  methods  than  the  right 
dominant  information  processing  style  subjects.  In  addition,  the  integrated  information  processing  style  subjects  used  significantly 
more  analytical  methods  than  did  the  right  dominant  information  processing  style  subjects. 

The  present  study  demonstrated  that  the  information  processing  style  seems  to  play  an  important  role  in  how  an  individual  interacts 
with  the  hypermedia  systems.  The  conclusion  should  be  confirmed  by  additional  research. 

Introduction 
Statement  of  the  Problem 

Hypermedia  systems,  used  as  sophisticated  databases  and  instructional  delivery  systems,  are  becoming  more 
powerful  and  hence  more  popular  in  educational  and  corporate  environments.  A review  of  the  literature  on 
hypermedia  systems  indicates  that  these  rapidly  evolving  systems  offer  two  unique  features  over  traditional  media: 

(a)  increased  richness  of  information  and  (b)  nonlinearity  of  information  structure.  Many  researchers  claimed  that 
the  overall  advantage  of  hypermedia  systems  with  these  two  unique  features  is  the  ability  to  accommodate  individual 
differences  in  various  users’  learning  styles  so  that  learning  can  be  more  effective  for  everyone  (Ayersman,  1993; 
Bermudez  & Palumbo,  1994;  Grice  & Ridgway,  1995;  Jonassen,  1988;  Larson,  1992).  Current  interest  in 
hypermedia  systems,  however,  has  focused  mainly  on  the  systems  implementation  and  software  development.  Little 
attention  has  been  paid  to  the  complexity  of  how  different  learning  style  users  actually  interact  with  non-linear 
hypermedia  systems  (Ayersman,  1996;  Beasley  & Waugh,  1997;  Harmon  & Dinsmore,  1994;  Savenye,  Leader, 

Jones,  Dwyer,  & Jiang,  1996). 

Many  research  studies,  on  the  other  hand,  have  focused  exclusively  on  analysis  of  general  users’ 
navigational  patterns  and  categorizing  navigational  patterns  in  hypermedia  systems  themselves.  These  studies 
indicate  that  there  are  considerable  variation  among  different  users  in  their  navigational  patterns,  including 
navigational  depth  which  reveals  how  comprehensively  users  access  the  system,  navigational  path  which  displays 
what  sequences  users  travel,  and  navigational  method  which  shows  what  navigational  means  users  employ  when 
using  the  system  (Barab,  et  al.,  1997;  Beasley  & Vila,  1992;  Canter  et  al.,  1985;  Homey,  1993;  Orey  & Nelson, 

1994;  Savenye,  1996;  Qiu,  1993).  However,  relatively  few  studies  have  investigated  further  to  identify  the  factors 
that  might  impact  diversity  of  navigational  patterns  among  different  users. 

The  information  processing  styles  implies  great  influence  on  variation  of  individuals’  navigational 
behaviors.  The  findings  of  separate  bodies  of  research  in  information  processing  styles  and  navigational  patterns 
imply  that  left-right  brain  dominant  information  processors  might  interact  quite  differently  within  nonlinear 
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hypermedia  systems  applying  different  navigational  strategies.  Unfortunately,  there  has  been  little  research  to  bring 
the  possible  implications  of  information  processing  styles  on  users’  interaction  related  to  hypermedia  systems. 

\ 

Purpose  of  the  Study 

The  purpose  of  this  study  was  to  investigate  the  relationships  between  users’  information  processing  styles 
and  navigational  patterns  within  hypermedia  systems.  This  study  investigated  individuals’  information  processing 
styles  and  examined  whether  information  processing  styles  is  related  to  users’  navigational  patterns  in  hypermedia 
systems.  Three  aspects  of  navigational  patterns  were  investigated:  (a)  navigational  depth  patterns  which  reveal  how 
comprehensively  users  access,  (b)  navigational  path  patterns  which  display  what  sequences  users  follow,  and  (c) 
navigational  method  patterns  which  show  what  methods  users  employ  when  using  the  system. 

Theoretical  Framework 
Nonlinearity  of  Hypermedia  Systems 

Traditional  computer-based  instruction  (CBI)  systems  are  rigid  in  their  information  structure  (Ellis  et  ah, 
1993).  The  learner  has  to  follow  a limited  number  of  predetermined  paths,  which  are  structured  usually  in  a linear 
fashion.  Therefore,  little  opportunity  exists  for  individual  choices  and  hence  everybody  has  to  use  the  system  in 
much  the  same  way. 

In  comparison,  hypermedia  systems  eliminate  the  constraints  of  sequential  presentation  of  information. 
Instead  of  organizing  and  presenting  information  sequentially  with  traditional  books  and  CBI  systems,  hypermedia 
systems  adapt  a different  concept  of  information  structure.  Information  in  a hypermedia  system  is  stored  as  small 
discrete  units  and  is  linked  together  into  complex  networks.  Unlike  traditional  information  systems,  which  have 
mostly  linear  structure,  hypermedia  systems  are  nonlinear. 

In  general,  nonlinear  hypermedia  systems  provide  users  much  greater  control  over  navigational  strategies 
than  any  other  traditional  media.  Nonlinear  hypermedia  systems  allow  users  to  use  personalized  information -see king 
strategies  and  hence  users  of  hypermedia  systems  can  make  their  own  decisions  about  navigational  strategies  based 
on  individuals’  preferences  to  accomplish  their  objectives  and/or  needs.  In  this  context,  researchers  claimed  that 
hypermedia  has  great  potential  to  address  individual  differences  in  users’  learning  styles  (Ayersman,  1993;  Rabbitt  & 
Miller,  1996;  Jonassen,  1988;  Knussen,  Kibby,  & Tanner,  1991;  Reed  & Oughton,  1997).  The  researchers  believed 
that  individuals  will  actively  interact  with  the  systems,  exploiting  the  navigational  advantage  of  the  nonlinear 
structure  according  to  users’  preference  for  learning  styles.  This  belief  requires  further  study. 

Present  State  of  Research  on  Navigational  Patterns  in  Hypermedia  Systems 

According  to  Ayersman  (1996)  and  Barab,  Bowdish,  and  Lawless  (1997),  few  studies  have  been  conducted 
to  investigate  the  nature  of  different  individual  users’  navigation  within  hypermedia  systems.  Reed  and  Oughton 
(1997)  and  Savenye  et  al.  (1996)  also  argued  that  little  empirical  research  pertaining  to  how  different  users  interact 
with  nonlinear  hypermedia  applications  has  been  conducted  to  date,  and  that  more  research  needs  to  be  done. 

Meanwhile,  many  other  research  has  focused  more  exclusively  on  categorizing  navigational  patterns  in 
hypermedia  systems  themselves.  The  navigational  patterns  found  in  the  literature  are  mainly  divided  into  three 
categories:  (a)  navigational  depth  patterns  which  reveal  how  comprehensively  users  access  the  system,  (b) 
navigational  path  patterns  which  display  what  sequences  users  follow,  and  (c)  navigational  method  patterns  which 
show  what  methods  users  employ  when  using  the  system. 

Researchers  examined  users’  navigational  depths  accessed  within  hypermedia  systems  (Canter,  Rivers,  & 
Storrs,  1985;  Schroeder  and  Grabowski,  1995).  Canter  et  al.  examined  navigational  depths  as  an  indication  of  the 
range  of  exploration  undertaken  by  users.  They  grouped  navigational  depth  patterns  into  three  different  ranges  as 
high,  medium,  and  low  exploration. 

Other  studies  examined  users’  general  navigational  path  patterns  in  hypermedia  systems.  Orey  and  Nelson 
(1994)  identified  four  categories  of  navigational  path  patterns:  rings,  loops,  paths,  and  spikes.  Similarly,  Homey 
(1993)  described  five  navigational  path  patterns  including  linear  traversal,  side  trip,  star,  extended  star,  and  chaotic. 
Although  these  researchers  identified  various  and  specific  navigational  path  patterns  with  hypermedia  systems,  other 
researchers  categorized  navigational  path  patterns  in  hypermedia  systems  more  generally,  as  linear  and  nonlinear 
path  patterns  (Beasley  & Vila,  1992;  McCluskey,  1993;  Parunak,  1989). 

Researchers  also  examined  prominent  navigational  methods  users  employed  when  using  the  system 
(Beasley  & Vila,  1992;  Dias  & Sousa,  1997;  Nielson,  1989;  Qiu,  1991,  1993;  Savenye  et  al,  1996).  Navigational 
methods  commonly  included  in  hypermedia  systems  consist  of  index,  keywords  search  functions,  maps,  hypertexts, 
menus,  and  tables  of  contents.  These  navigational  method  patterns  were  categorized  into  two  groups,  analytical  and 
holistic  methods,  based  on  the  property  of  the  navigational  methods  (Campagnoni  & Ehrilch,  1989;  Marchionini  & 
Shneiderman,  1988;  Qiu,  1993).  According  to  Marchionini  and  Shneiderman  (1988),  analytical  methods  are  direct 
navigational  means  which  involve  a well -structured  search  for  specific  information,  while  holistic  methods  are  more 
likely  indirect  means  which  involve  a relatively  unstructured  search  and  assist  users’  searching  process  indirectly. 
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The  overall  conclusion  of  those  studies  on  navigational  patterns  is  that  there  is  a great  deal  of  diversity  in 
navigational  patterns  which  individuals  employ,  including  navigational  depth,  path,  and  method  patterns.  In 
addition,  it  is  widely  accepted  that  the  diversity  of  navigational  patterns  results  from  individuals’  travel  within 
hypermedia  systems  using  different  navigational  strategies.  Most  of  the  studies,  however,  did  not  investigate  further 
why  different  users  rely  on  so  many  different  navigational  strategies  when  utilizing  hypermedia  systems. 

Furthermore,  few  of  these  studies  considered  individuals’  learning  styles  that  might  impact  on  users’  navigational 
strategies. 

Learning  Styles  and  Information  Processing  Styles 

The  term  “learning  styles  ’’has  been  used  interchangeably  with  “cognitive  styles,”  “brain  dominance,” 
“information  processing  styles,”  and  “learning  modalities”  by  many  (Ayersman,  1993  & 1996;  Entwistle,  1981;  Liu 
& Reed,  1994;  Marrison  & Frick,  1994;  Shaughnessy,  1996). 

Learning  style  has  long  been  considered  a fundamental  element  in  individual  differences,  which  impact  on 
problem  solving  strategies  in  various  learning  situations  (Ayersman,  1993;  Carbo,  Dunn,  & Dunn,  1986;  Hanson, 
1996;  Leader  & Klein,  1996;  Wilson,  1988).  Numerous  models  for  learning  styles  have  been  developed.  The 
models  are  slightly  different  based  on  which  cognitive  variables  each  model  stresses  in  the  learning  process. 

Witkin’s  Cognitive  Styles  (Witkin,  1950)  focuses  on  how  learners  perceive  information  in  terms  of  learners’ 
dependency  and  independency  on  a surrounding  learning  environment.  Kolb’s  Learning  Styles  (Kolb,  1985)  deals 
more  with  learners’  general  attitudes  in  the  learning  process,  such  as  converger,  diverger,  assimilator,  and 
accommodator.  Sensory  Modalities  (Sprinthall  & Sprinthall,  1990)  focuses  on  individual  preferences  for  sensory 
input  of  information  as  visual  learners  and  auditory  learners.  Brain  hemisphericity  (Sonnier,  1992,  96)  deals  with 
specific  problem  solving  strategies  in  performing  cognitive  tasks  based  on  hemispheric  processing  preference  style. 

The  information  processing  styles  based  on  brain  hemisphericity  theory  focuses  particularly  on  individuals’ 
differences  in  overall  information  processing  strategies  for  problem  solving  based  on  hemispheric  information 
processing  preference  (Gardner,  1985;  Hunter,  1976;  Sonnier,  1992;  Torrance,  1982;  Torrance  & Taggart,  1984). 
The  information  processing  style  model  examines  closely  the  cognitive  strategies  of  how  individuals  approach 
problem  solving  differently  in  various  learning  situations.  In  this  context,  information  processing  style  model 
suggests  greater  implications  on  users’  navigational  strategies  than  any  other  learning  style  models  described,  since 
navigational  patterns  are  representations  of  the  strategies  used  by  individuals  to  process  information  in  nonlinear 
hypermedia  systems. 

Functional  lateralization  of  the  brain  has  been  investigated  in  efforts  to  explain  the  differences  in  the 
abilities  of  the  two  halves  of  the  brain.  There  has  been  much  evidence  indicating  that  there  are  areas  of 
specialization  giving  each  hemisphere  dominance  in  controlling  particular  cognitive  functions.  The  overall 
indications  in  related  literature  are  that  the  left  hemisphere  is  predominantly  involved  with  analytic  thinking, 
specifically,  language  and  logic.  The  right  hemisphere,  one  the  other  hand,  is  involved  primarily  with  holistic 
thinking,  responsible  for  orientation  in  space,  artistic  talents,  and  bodily  awareness  (Gardner,  1985;  Hellige,  1993; 
Iaccino,  1993;  Springer  & Deusch,  1989;  Taggart  & Torrance,  1984). 

In  addition,  human  beings  are  thought  to  vary  in  terms  of  their  cerebral  dominance.  The  notion  of 
hemispheric  dominance  among  people  is  that  the  individual  has  an  overall  preference  for  using  cognitive  ability  of 
one  side  of  the  brain  for  problem  solving.  Numerous  studies  support  the  notion  of  preferred  style  in  hemispheric 
processing  preference  among  individuals  (Herrmann,  1981;  Shaughnessy,  1996;  Torrance  et  al.,  1984;  Zaidel, 

Clarke,  & Suyenobu,  1990).  These  studies  suggest  that  left-dominant  processors  usually  prefer  structured  tasks, 
approach  tasks  systematically,  use  analytical  strategies,  solve  problems  logically,  and  are  more  likely  to  process 
information  in  linear  fashion.  In  contrast,  right-dominant  processors  tend  to  prefer  open-ended  tasks,  approach  tasks 
by  exploration,  use  holistic  strategies,  solve  problems  intuitively,  and  process  information  in  nonlinear  ways. 

Implications  of  Information  Processing  Styles  on  Users'  Navigation  Patterns 

Findings  of  research  in  brain  hemisphericity  theory  suggest  that  each  side  of  the  brain  is  dominant  in 
controlling  particular  cognitive  functions  for  problem  solving  and  that  human  beings  vary  in  terms  of  their  cerebral 
dominance.  Many  studies  investigated  the  effects  of  individuals’  cerebral  dominance  on  problem  solving  strategies 
and  these  studies  indicated  that  cerebral  dominance  impacts  problem  solving  strategies  among  people  in  various 
learning  environments  (Herrmann,  1981;  Hough,  1987;  Robey  & Taggart,  1982;  Shaughnessy,  1996;  Zaidel  et  al., 
1990). 

Information  processing  styles  categorized  according  to  brain  hemisphericity  theory  might  inpact  on  users’ 
navigational  patterns  within  hypermedia  systems.  That  is  because  navigational  patterns  are  representations  of  the 
strategies  used  by  individuals  to  process  information  in  nonlinear  hypermedia  systems  much  the  same  as  with 
individuals’  problem  solving  strategies  in  various  other  learning  environments.  Then,  there  might  be  relationships 
between  information  processing  styles  among  people  and  navigational  strategies  they  employ  within  hypermedia 
systems. 
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On  the  basis  of  the  findings  of  research  in  information  processing  styles  and  navigational  patterns,  it  is 
implied  that  left  brain  dominant  information  processors  might  prefer  to  explore  hypermedia  systems  in  a relatively 
limited  depth  while  right  brain  dominant  information  processors  might  tend  to  explore  with  hypermedia  systems  in  a 
relatively  comprehensive  depth  pattern.  Left  brain  dominant  information  processors  might  prefer  to  access  nodes  in 
much  more  linear  path  patterns  while  right  brain  dominant  information  processors  might  be  inclined  to  interact  with 
hypermedia  systems  in  relatively  nonlinear  path  patterns.  Similarly,  left  brain  dominant  information  processors 
might  prefer  to  navigate  within  hypermedia  systems  using  more  analytical  navigational  methods  such  as  using  index 
and  keyword  functions.  In  contrast,  right  brain  dominant  information  processors  might  travel  hypermedia  systems 
using  more  holistic  navigational  features  such  as  map,  hypertext,  and  table  of  contents. 

Research  Questions 

The  general  research  questions  of  this  study  were  as  follows: 


1.  What  are  the  relationships  between  hypermedia  users’  information  processing  styles  and  their  navigational  depth 
patterns? 

2.  What  are  the  relationships  between  hypermedia  users’  information  processing  styles  and  their  navigational  path 
patterns? 

3.  What  are  the  relationships  between  hypermedia  users’  information  processing  styles  and  their  navigational  method 
patterns? 


Significance  of  the  Study 

Results  of  the  study  would  provide  insights  into  the  relationships  between  navigational  patterns  and 
information  processing  styles  of  hypermedia  users.  These  results  then  would  provide  useful  information  for  advising 
learners  regarding  selecting  more  efficient  and  effective  navigational  patterns  based  on  their  information  processing 
styles.  In  addition,  the  detailed  information  of  users'  navigational  behaviors  could  provide  further  guidance  for 
designing  hypermedia  applications  that  are  more  effective  and  more  user-centered.  The  results  of  this  study  could 
also  provide  additional  information  that  would  verify  the  effectiveness  of  hypermedia  systems  in  terms  of  addressing 
individual  differences. 

Method 

Operational  Definitions 

Hypermedia  Environment:  The  A.g.i.l.e.  Trainer™  hypermedia  program  developed  by  Knowledge 
Solutions  Inc.  Bethlehem,  PA  (Harvey  & Nelson,  1995). 

The  A.g.i.l.e.  Trainer™  includes  a more  flexible  navigational  structure  that  uses  pull-down  menus  than 
other  programs  in  which  “next"  and  “previous"  buttons  are  the  most  obvious  navigational  means  to  travel  the 
system.  Refer  to  Figure  1 through  Figure  3 for  basic  screen  layouts  of  three  sets  of  pull-down  menus  in  the  A.g.i.l.e. 
Trainer™. 

It  is  important  to  note  that  lists  of  resources  in  the  pull-down  menus  represent  the  available  nodes  in  the 
experimental  system.  The  researcher  used  the  term  node  when  referring  to  an  information  resource  in  the  system. 
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Figure  1.  Pull-Down  Menu  in  “ Course  " Interface 
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Figure  2.  Pull-Down  Menu  in  “ Search  ” Interface 
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Figure  3.  Pull-Down  Menu  in  “Index”  Interface 


Left , Right , and  Integrated  Information  Processors: 

Three  different  styles  of  information  processors  categorized  by  the  Human  Information  Processing  Survey 
(HIPS)  (Torrance  et  al.,  1984). 

It  should  be  noted  that  the  HIPS  categorizes  four  different  information  processors:  (a)  left,  (b)  right,  (c) 
integrated,  and  (d)  mixed.  The  mixed  information  processors,  however,  was  excluded  from  this  study.  The  reason 
for  excluding  this  group  was  that  mixed  information  processors  typically  use  either  left  or  right  brain  dominant 
strategies  at  random  and  therefore  this  group  of  people  was  not  likely  to  show  any  dominant  navigational  patterns. 


Navigational  Depth 

Subjects’  scores  on  exploration  value,  which  is  adapted  from  Canter  et  al.  (1985): 


Exploration  value  = 


The  number  of  different  nodes  visited 


The  total  number  of  nodes  available  in  the  system 


The  exploration  value  ranges  from  0 to  1.  The  higher  the  exploration  value,  the  more  comprehensive  the 
navigational  depth  pattern  is  and  the  lower  the  exploration  value,  the  more  limited  the  navigational  depth  pattern  is  in 
the  current  study. 

Depth  refers  to  the  degree  of  users’  exploration  by  accessing  nodes  both  in  different  levels  and  across  the 
same  level  in  the  system  structure.  It  should  be  noted  that  nodes  visited  repeatedly  are  counted  only  once  in  the 
current  study.  That  is  because  the  objective  here  is  to  examine  how  comprehensively  users  explore  various  nodes 
available  in  the  system.  The  performance  of  subjects  who  take  repeated  visits  to  the  same  node  will  be  analyzed  in  a 
follow-up  study. 


Navigational  Path 

Subjects’  scores  on  linearity  value,  which  is  defined  as: 

The  number  of  linear  navigational  links  followed 

Linearity  value  = ; 

Total  number  of  either  linear  or  nonlinear 
navigational  links  followed 


A linear  link  is  defined  as  a link  between  two  nodes  connecting  a node  to  a subsequent  node  that  is  listed 
adjacent  to  the  first  node  chosen  in  a pull-down  menu.  A navigational  link  between  two  nodes  connecting  a node  to 
the  previous  node  visited  using  “Back”  button  is  also  a linear  link.  A nonlinear  link  is  defined  as  a link  between  two 
nodes  connecting  a node  to  a subsequent  node  that  is  not  listed  adjacent  to  the  first  node  chosen  in  a pull-down 
menu.  In  addition,  a link  between  two  nodes  connecting  a node  in  a navigational  method  to  a subsequent  node  in 
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another  method  is  a nonlinear  link.  Any  navigational  links  connecting  two  nodes  by  clicking  hypertexts  were  also 
considered  as  nonlinear  links. 

The  linearity  value  ranges  firomO  to  1.  The  higher  the  linearity  value,  the  more  linear  the  navigational  path 
pattern  is  and  the  lower  the  linearity  value,  the  more  nonlinear  the  navigational  path  pattern  is  in  the  current  study. 

The  pull-down  menus  display  the  list  of  nodes  available  in  the  system  and  users  can  access  any  information 
in  the  list  at  any  time  in  any  sequence.  Users  can  navigate  the  system  in  either  a linear  or  nonlinear  sequence  simply 
by  choosing  nodes  which  are  listed  adjacent  to  each  other  or  which  are  not  listed  adjacent  to  each  other,  respectively, 
as  indicated  in  the  pull-down  menus. 


Navigational  Method 

Subjects'  scores  on  analytical  methods  value,  which  is  adapted  from  Qiu  (1993): 


Analytical  methods  value  = 


The  number  of  analytical  methods  employed 


Total  number  of  navigational  methods  employed 
by  analytical,  holistic,  or  mixed  methods. 


Analytical  methods  are  defined  as  navigational  means  that  lead  a direct  and  well-structured  search  for  a 
specific  information  sought.  Analytical  methods  include  search  (keyword  search)  and  index.  Holistic  methods  are 
defined  as  indirect  and  less-structured  navigational  means  that  assist  users  indirectly  to  find  information  sought. 
Holistic  methods  include  course  (table  of  contents),  help,  back,  done  searching,  and  hypertext. 

An  analytical  or  a holistic  method  were  counted  each  time  one  of  the  method  is  used  to  create  a link 
between  two  nodes.  A mixed  method  was  counted  each  time  combinations  of  analytical  and  holistic  methods  are 
used  to  create  a link  between  two  nodes. 

Analytical  methods  value  ranges  from  0 to  1.  The  higher  the  analytical  methods  value,  the  more  analytical 
the  navigational  method  pattern  is  and  the  lower  the  analytical  methods  value,  the  more  holistic  the  navigational 
method  pattern  is  in  the  current  study. 

The  A.g.i.l.e.  Trainer  ™ program  supports  the  following  navigational  methods  summarized  in  Table  1. 
Table  1.  Types  of  Navigational  Methods  in  A.g.i.l.e.  Trainer™  Program 


Methods 

Description 

Analytical- 

Holistic 

Back 

To  return  to  the  node  user  most  recently  visited. 

Holistic 

Course 

To  bring  an  interface  which  allows  user  to  search  information  using  a 
pull-down  menu. 

Holistic 

Search 

To  bring  an  interface  which  allows  user  to  search  specific  information  by 
typing  keywords. 

Analytical 

Done  Searching 

To  return  to  the  course  interface  from  search  interface. 

Holistic 

Help 

To  go  to  a help  screen  about  general  guidelines  for  navigating  the 
system. 

Holistic 

Index 

To  bring  an  interface  that  allows  user  to  search  specific  information 
presented  in  index. 

Analytical 

Hypertext 

To  go  another  node  which  describes  the  content  about  the  hypertext  by 
jumping  to  a node  out  of  sequence  in  the  system. 

Holistic 

Sample 


The  sample  subjects  were  102  undergraduate  students  enrolled  in  management  courses  in  the  College  of 
Business  and  Economics  at  Lehigh  University.  Participation  was  voluntary.  To  ensure  maximum  participation, 
incentives  in  the  form  of  bonus  points  were  offered  in  the  courses  in  which  the  subjects  were  enrolled.  Thirty  four 
students  were  selected  for  each  type  of  hemispheric  processing  preference  style  (left,  right,  or  integrated). 
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Instruments 

Human  Information  Processing  Survey 

Human  Information  Processing  Survey  (HIPS)  is  a tool  for  assessing  individuals  in  terms  of  hemispheric 
processing  preference  (Taggart  & Torrance,  1984).  This  survey  yields  four  possible  categories  of  information 
processors:  (a)  left  dominant,  (b)  right  dominant,  (c)  mixed,  and  (d)  integrated  information  processors.  According  to 
Taggart  and  Torrance,  left  brain  dominant  processors  think  in  predominantly  systematic  ways,  approach  tasks 
sequentially,  and  solve  problems  analytically.  On  the  other  hand,  right  brain  dominant  processors  are  believed  to 
adapt  primarily  the  opposite  strategies  in  problem  solving  such  that  they  think  in  an  exploratory  manner,  approach 
tasks  non-sequentially,  and  solve  problems  in  holistic  ways.  Integrated  information  processors  are  individuals  who 
use  simultaneously  the  left  and  right  dominant  strategies  in  proportion  appropriate  for  the  situation  while  mixed 
information  processors  are  individuals  who  use  either  a left  dominant  or  a right  dominant  strategy  at  random. 

Open-Ended  Questions  and  Closed-Ended  Questions 

The  researcher  designed  two  types  of  computer  task  questions,  3 items  for  open-ended  tasks  and  5 items  for 
closed-ended  tasks  in  order  to  assist  users  in  using  the  experiment  program  purposely  and  comprehensively.  Open- 
ended  tasks  in  this  research  required  users  to  respond  in  general  terms  based  on  understanding  the  overall  content  of 
the  hypermedia  system.  Close-ended  tasks  in  this  research  required  users  to  respond  in  specific  terms  that  can  be 
found  in  the  hypermedia  system. 

The  questions  were  reviewed  by  the  developer  of  the  A.g.i.l.e  Resource  ™ hypermedia  program.  Dr. 

Nelson,  Vice  President  for  Development,  Knowledge  Solution  Inc.  to  determine  whether  questions  adequately 
address  the  intention  for  leading  users  to  use  the  program  purposely  and  comprehensively.  He  agreed  that  the 
questions  adequately  address  the  purpose.  It  was  not  necessary  to  make  any  changes  to  the  instrument.  No  formal 
statistical  analysis  to  determine  reliability  of  the  instrument  was  performed  since  the  questions  are  a part  of  treatment 
to  assist  the  subjects  in  using  the  experiment  program  purposely. 

Subject  Demographic  Information  Questionnaire 

The  researcher  designed  a subject  demographic  information  questionnaire.  The  questionnaire  was  used  to 
collect  demographic  data  on  the  sample  in  order  to  determine  gender,  age,  year  in  college,  previous  computer 
training,  and  computer  expertise  of  each  subject. 

Demographic  data  were  used  to  determine  whether  there  were  initial  differences  other  than  brain 
hemisphericity  among  three  information  processor  groups.  No  formal  statistical  analysis  to  determine  validity  and 
reliability  of  the  instrument  was  performed  since  the  questionnaire  was  not  for  testing  any  effects.  Rather,  the 
questionnaire  was  for  collecting  demographic  information  of  the  subjects. 

Research  Design 

The  causal-comparative  research  design  was  used  to  explore  the  relationships  between  hypermedia  users’ 
information  processing  styles  and  their  navigational  patterns.  The  causal-comparative  research  method  was  selected 
instead  of  the  experimental  method  because  it  allows  the  researcher  to  study  the  relationships  between  variables 
under  a condition  where  experimental  manipulation  is  not  possible.  It  was  impossible  for  the  researcher  to 
manipulate  hypermedia  users’  information  processing  styles  and  their  navigational  behaviors  in  using  the  program. 
The  hypermedia  program  were  also  given  as  a study  environment  rather  than  as  an  experimental  treatment 
manipulated  for  the  study. 

The  independent  variable  in  the  present  study  was  the  informational  processing  styles  of  the  users.  It  had 
three  categories:  (a)  left  dominant  information  processors,  (2)  right  dominant  information  processors,  and  (c) 
integrated  information  processors.  The  dependent  variables  were  the  three  different  navigational  patterns:  (a) 
navigational  depth  patterns,  (b)  navigational  path  patterns,  and  (c)  navigational  method  patterns. 

Three  separate  one-way  analysis  of  variance  (ANOVA)  were  used  to  test  the  three  hypotheses  stated 
individually.  ANOVA  was  selected  to  test  the  null  hypotheses  because  the  study  involves  comparison  of  three 
groups.  The  researcher  also  judged  that  three  separate  one-way  ANOVA  would  be  appropriate  to  test  the  hypotheses 
since  the  study  had  one  categorical  independent  variable  and  three  measured  dependent  variables  which  were  not 
related  to  one  another  (Borg  & Gall,  1989). 

The  Statistical  Package  for  Social  Science  (SPSS  Inc.,  1995)  was  used  to  perform  the  ANOVA  procedures. 
An  alpha  level  of  .05  was  set  as  the  criterion  for  rejection  of  the  null  hypothesis.  According  to  Borg  and  Gall  (1989), 
the  .05  level  is  the  most  commonly  chosen  alpha  level  for  rejecting  the  null  hypothesis  in  educational  research. 

A power  level  of  .70  assuming  a medium  effect  size  of  .25  was  used  throughout  the  study.  Cohen  (1969) 
reported  that  .80  serves  as  a convention  for  desired  power  level. 
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Pilot  Study 

A pilot  study  was  conducted  to  make  sure  that  the  procedures  and  the  instruments  to  be  used  in  the  study 
operate  properly.  The  main  objective  of  the  pilot  study  was  to  test  the  hypermedia  system  and  computer  utility  that 
records  users*  interactions  within  the  hypermedia  system.  At  the  conclusion  of  the  pilot  study,  procedures  and 
instruments  were  examined  by  the  researcher  to  determine  if  they  performed  satisfactorily.  Some  minor  changes 
were  made  according  to  the  results  of  the  pilot  study. 

Research  Study 

Hypermedia  Program  - A.g.LLe.  Trainer™ 

The  A.g.i.l.e.  Trainer™  is  an  interactive  multimedia  CD-ROM  knowledge  base  about  organizational  agility 
and  examples  of  agile  practices.  The  A.g.i.l.e.  Trainer™  is  developed  by  Knowledge  Solutions  Inc.,  Bethlehem,  PA 
(Harvey  & Nelson,  1995).  The  program  is  designed  as  a reference  tool  or  self-teaching  module.  It  is  aimed  toward 
various  levels  of  users  in  terms  of  computer  experience  from  novice  computer  users  to  expert  computer  users. 

The  A.g.i.l.e.  Trainer  ™ contains  various  types  of  information  including  text,  graphics,  sound,  video,  and 
animated  segments  that  illustrate  principles  and  provide  examples  about  agility.  The  A.g.i.l.e.  Trainer™  provides 
hundreds  of  resources  about  agility  and  introduces  more  than  175  companies  as  examples  of  high-performance  in 
business.  The  A.g.i.l.e.  Trainer™  has  powerful  features  that  are  easy  to  use.  Hundreds  of  information-providing 
text  screens  are  electronically  linked  and  the  relationships  among  them  are  apparent. 

The  A.g.i.l.e.  Trainer™  requires  a 386  or  higher  IBM  compatible  computer,  256  color  VGA  monitor,  MPC 
compatible  sound  card  and  speakers,  at  least  4MB  RAM,  5MB  HD  space,  and  Windows  3.1  or  later. 

Search  Tasks 

Previous  studies  indicated  that  the  search  task  is  a major  factor  that  influences  navigational  patterns  in 
general  (Campagnoni  & Ehrlich,  1989;  Nielson,  1989;  Qiu,  1993).  For  instance,  an  open-ended  task  leads  users  to 
navigate  the  hypermedia  system  using  holistic  methods  while  a closed-ended  task  leads  users  to  employ  more 
analytical  methods.  The  current  research,  however,  attempted  to  determine  the  relationship  between  users* 
information  processing  styles  and  navigational  patterns  within  hypermedia  system  regardless  of  search  tasks.  The 
researcher,  therefore,  provided  two  types  of  tasks,  open-ended  tasks  and  closed-ended  tasks  for  all  users,  so  that 
users  would  navigate  the  hypermedia  program  for  identical  tasks. 


Procedures 

The  research  study  administered  by  the  researcher  and  an  assistant  throughout  most  of  data  collection 
process.  The  assistant  trained  to  help  the  research  process  only  as  directed  by  the  researcher  in  order  to  avoid  any 
risk  of  having  the  results  affected  by  the  assistant.  The  assistant  was  not  allowed  to  respond  to  any  requests  or 
questions  directly  related  to  the  research  study.  Rather,  the  assistant  helped  only  for  mechanical  tasks  such  as 
distributing  materials,  collecting  materials,  installing  the  A.g.i.l.e.  Trainer™  program,  trouble  shooting,  and  other 
activities  that  would  not  affect  the  results. 

The  research  study  was  divided  into  two  sessions,  session  A and  session  B.  Session  A was  for  drawing 
sample  subjects  and  session  B was  for  conducting  the  main  experiment  with  the  hypermedia  program. 

In  session  A the  researcher  visited  three  management  classes,  MGT  269(1),  MGT269  (2),  and  MGT  270,  in 
the  College  of  Business  and  Economics.  The  combined  enrollment  of  these  classes  was  approximately  200  students 
with  about  65  students  for  each  class.  Professors  of  each  class.  Dr.  Trent  and  Dr.  Poole,  agreed  to  have  their 
students  participate  in  the  study  and  to  offer  incentives  in  the  form  of  extra  credit  in  the  courses  that  the  subjects 
were  enrolled.  The  researcher  visited  the  three  management  classes  at  the  end  of  each  class.  Each  student  was  given 
a brief  explanation  of  what  was  required  of  him  or  her  in  the  experiment  and  a consent  form  was  signed.  The 
subjects  were  then  be  asked  to  complete  the  Human  Information  Processing  Survey  (HIPS)  which  is  a pencil  and 
paper  self-report  inventory.  The  survey  took  20  to  30  minutes. 

Upon  completion  of  the  HIPS,  the  researcher  scored  the  responses  and  identified  subjects’  hemispheric 
processing  styles  following  the  instructions  in  the  HIPS,  the  researcher  selected  randomly  among  the  participants 
approximately  40  subjects,  using  a lottery  system  for  each  type  of  hemispheric  processing  preference  style  (left, 
right,  or  integrated)  for  the  experiment.  The  selected  subjects  then  were  asked  to  choose  a time  slot  on  the  schedule 
arranged  for  the  main  experiment.  Experiment  times  were  scheduled  at  least  twice  a day  for  five  days,  one  AM  and 
one  PM,  for  an  hour  so  that  subjects  could  participate  at  times  that  were  convenient  for  them. 

In  session  B subjects  were  assigned  identification  numbers  randomly  by  following  a lottery  system.  These 
assigned  numbers  were  used  on  all  tests  and  forms  to  be  completed  by  the  subjects.  The  master  list  of  assigned 
numbers  were  maintained  by  the  researcher. 

Subject  demographic  information  questionnaires  were  administered  to  all  subjects  before  they  participated 
the  main  experiment  with  the  hypermedia  program.  The  directions  for  how  to  use  the  hypermedia  system  and  the 
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main  experimental  procedure  were  given  using  a videotape  recording  in  order  to  keep  the  consistency  among 
subjects  who  participated  in  the  experiment  at  different  experiment  times.  Prior  to  conducting  the  formal  experiment 
with  the  program  the  videotape  provided  a short  demonstration  to  the  subjects  about  the  experimental  system 
including  the  overall  structure  of  the  hypermedia  system  and  how  to  navigate  within  the  system.  All  subjects  had  an 
opportunity  to  try  out  the  system  until  they  feel  confident  about  using  it. 

The  subjects  then  were  asked  to  perform  the  two  tasks.  The  first  task  was  to  answer  the  open-ended 
questions  based  on  the  content  in  the  hypermedia  system.  The  subjects  were  asked  to  browse  the  hypermedia 
system,  A.g.i.l.e.  Trainer™,  to  get  familiar  with  the  concept  of  Agility  and  to  answer  the  three  general  questions  in 
the  work  sheet.  A copy  of  the  open-ended  questions  were  given  to  the  subject  after  subjects  review  the  program  as 
they  might  wish.  The  researcher  was  not  be  able  to  answer  any  questions  the  subjects  might  have  while  subjects 
interacted  with  the  experimental  program,  other  than  making  sure  equipment  was  operating  correctly.  The  users 
interaction  activities  with  the  hypermedia  system  were  recorded  using  a Window  environment  utility  called 
‘Recorder.’  The  Recorder  recorded  user  interactions  simultaneously  while  users  interacted  with  the  hypermedia 
system.  The  computer  files  of  the  recording  results  were  saved  by  the  researcher  and  the  assistant  when  the  subject 
indicated  he  or  she  was  finished. 

Next,  the  subjects  were  asked  to  perform  the  second  task  which  were  closed-ended  questions  to  answer 
based  on  specific  information  that  could  be  found  in  the  hypermedia  system.  The  researcher  did  not  answer  any 
questions  the  subjects  might  have  while  subjects  interacted  with  the  experimental  program,  other  than  making  sure 
equipment  was  operating  correctly.  The  users’  interaction  activities  with  the  hypermedia  system  were  again  recorded 
using  a Windows  environment  utility  called  ‘Recorder.’  The  Recorder  was  running  while  users  interacted  with 
hypermedia  system  at  the  same  time.  The  computer  files  of  the  recording  results  were  saved  by  the  researcher  and  the 
assistant  when  the  subject  indicated  he  or  she  was  finished. 

Data  Collection 

Details  of  each  user’s  interactions  with  the  hypermedia  program  were  recorded  using  the  ‘Recorder’  utility 
for  Windows  (Windows  3.1,  1993).  The  ‘Recorder’  automatically  recorded  and  saved  as  a movie  file  exactly  what 
users  did  and  how  they  interacted  with  the  program. 

The  measurement  of  the  variables  of  the  study  required  the  development  of  a data  processing  procedure 
specifically  for  the  study  since  the  study  uses  a non-standard  (self-developed)  measure  (Borg  & Gall,  1989). 

The  researcher  applied  guidelines  on  data  processing  procedures  from  several  research  handbooks  (Borg  & 
Gall,  1989;  Kidder,  81;  Livingston,  1988;  Savenye  & Robinson,  1996)  to  develop  valid  and  reliable  data  processing 
procedures.  The  data  processing  procedures  in  the  study  were  as  follows: 

Data  reduction  procedures.  The  original  data  were  collected  using  the  ‘Recorder’  program.  The  ‘Recorder’ 
recorded  exactly  what  users  did  and  how  users  interacted  with  the  experimental  environment  program  saving  the 
result  as  a recorder  file  just  as  the  same  format  as  a videotape  recording.  A data  reduction  process  was  needed  to 
convert  the  original  raw  data  from  recorder  files  format  to  a written  document  format  in  order  to  simplify  the  process 
of  representation  and  analysis  of  the  data  for  the  study  (Kidder,  1981).  One  third-party  reviewer  reviewed  the 
“Recorder”  files  and  recorded  subjects’  interactions  within  the  experimental  program  in  a data  reduction  form  in 
order  to  make  the  data  ready  for  data  coding  process. 

Data  coding  procedure.  The  researcher  developed  specific  data  coding  rules  for  the  study  that  included  a 
set  of  specific  steps  to  follow  for  data  coding.  An  external  evaluator  reviewed  the  protocol  of  data  coding  rules  in 
order  to  validate  the  rules  for  the  study.  The  evaluator  checked  the  coding  rules  to  ensure  clarity  and  the 
appropriateness  of  information  so  that  raters  could  make  decisions  based  on  the  coding  rules. 

The  researcher  refined  the  coding  rules  based  on  the  evaluator’s  suggestions.  When  the  final  coding  rules 
were  ready,  two  external  raters  coded  the  data,  following  the  coding  rules.  Two  separate  external  raters  coded  the 
entire  data  set  in  order  to  yield  more  valid  and  more  reliable  results  than  would  be  produced  by  one  rater  alone.  The 
coding  rules  were  fully  tested  with  the  two  raters  before  the  raters  actually  applied  the  rules  to  code  the  data. 

After  coding  trials  indicated  that  the  raters  yielded  satisfactory  codes  for  the  same  data,  each  rater  began  to 
code  separately  the  entire  subjects’  data.  The  researcher  examined  the  results  of  the  raters’  coding  data  and  resolved 
any  disagreements  in  coding  between  the  two  coders. 

Results 

Descriptive  Analysis 
Demographic  Data 

The  demographic  data  were  obtained  to  determine  whether  there  were  initial  differences  other  than 
information  processing  styles  among  three  information  processor  groups.  A demographic  information  questionnaire 
designed  by  the  researcher  was  used  to  obtain  complete  descriptions  of  the  subjects  in  three  groups. 
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The  researcher  found  strong  similarities  in  the  distribution  of  subjects  among  three  groups  in  gender,  age, 
years  in  college,  previous  computer  training,  and  computer  expertise.  This  demonstrates  the  benefits  of  drawing 
subjects  from  a large  population  group.  The  researcher  used  formal  tests  to  verify  that  the  groups  were  equivalent 
and  represented  the  data  in  a later  section  of  analysis  of  demographic  data. 

Analysis  of  Navigational  Pattern 

Navigational  depth.  Navigational  depth  is  a measure  reflecting  how  many  new  nodes  a user  visits  when 
navigating  in  a hypermedia  system.  In  this  study,  navigational  depth  was  measured  by  the  subjects’  scores  on 
exploration  value,  which  was  defined  as  the  ratio  of  the  number  of  different  nodes  visited  to  the  total  number  of 
nodes  available  in  the  system.  The  exploration  value  will  range  from  0 to  1. 


Table  2.  Means  and  Standard  Deviations  for  Navigational  Depth 


Groups 

n 

M 

SD 

S.  Error 

Left  Information  Processors 

34 

.0489 

.0125 

.0021 

Right  Information  Processors 

34 

.0581 

.0184 

.0032 

Integrated  Information  Processors 

34 

.0493 

.0133 

.0023 

Navigational  path.  Navigational  path  is  a measure  reflecting  which  sequences  a user  travels  when 
navigating  in  a hypermedia  system.  Navigational  path  was  measured  by  subjects’  scores  on  linearity  value,  which 
was  defined  as  the  ratio  of  the  number  of  number  of  linear  navigational  paths  followed  to  the  total  number  of  either 
linear  or  non-linear  navigational  paths  followed.  The  linearity  value  ranges  from  0 to  1. 


Table  3.  Means  and  Standard  Deviations  for  Navigational  Path 


Groups 

n 

M 

SD 

S.  Error 

Left  Information  Processors 

34 

.3528 

.1301 

.0223 

Right  Information  Processors 

34 

.4076 

1244 

.0213 

Integrated  Information  Processors 

34 

.3325 

.0722 

.0124 

Navigational  method. 

Navigational  method  is  a measure  reflecting  which  methods  users  employ  when 

navigating  in  a hypermedia  system. 

The  score  was  measured  by  the  analytical  methods  value,  which  was  defined  as 

the  ratio  of  the  number  of  analytical  methods  employed  to  the  total  number  of  either  analytical  or  holistic 

navigational  methods  employed.  Analytical  methods  value  ranges  from  0 to  1. 

Table  4.  Means  and  Standard  Deviations  for  Navigational  Method 

Groups 

n 

M 

SD 

S.  Error 

Left  Information  Processors 

34 

.4048 

.2271 

.0390 

Right  Information  Processors 

34 

.2751 

.1717 

.0294 

Integrated  Information  Processors 

34 

.2797 

.1665 

.0286 

A graphic  comparison  of  the  mean  scores  for  navigational  depth,  path,  and  methods  among  left  dominant 
information  processors,  right  dominant  information  processors,  and  integrated  information  processors  is  presented  in 
Figure  4. 
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Figure  4.  Mean  Scores  for  Navigational  Patterns  among  Left  Dominant  Information  Processors, 


Depth  Path  Method 

Navigational  Patterns 


Right  Dominant  Information  Processors,  and  Integrated  Information  Processors 
Inferential  Analysis 

Three  separate  one-way  ANOVA's  were  performed  to  determine  whether  there  were  significant  mean 
differences  in  three  different  navigational  patterns  (navigational  depth,  path,  and  method)  among  left  dominant 
information  processors,  right  dominant  information  processors,  and  integrated  information  processors. 

The  independent  variable  in  the  present  study  was  the  information  processing  style  of  subjects,  which  had 
three  levels:  left,  right,  and  integrated  information  processors.  The  dependent  variables  were  the  three  different 
navigational  patterns:  (a)  navigational  depth  patterns,  (b)  navigational  path  patterns,  and  (c)  navigational  method 
patterns. 

Analysis  of  Demographic  Data 

One-factor  ANOVA's  were  performed  to  determine  whether  there  were  initial  differences  between  groups 
on  several  variables.  There  was  no  significant  difference  between  groups  on  gender  (F  (2  ,90  ) = .770,  p >.05),  age 
(F  (2 , 90)  a .132  , p >.05),  and  year  in  college  (F  ( 2, 90)  = .125,  p >.05).  There  was  also  no  significant  difference 
between  groups  on  previous  computer  training  (F  (2 , 90  ) = 1.998  , p >.05)  and  computer  expertise  (F  (2 , 90)  = 
1.573,  p >.05). 

Information  Processing  Style  and  Navigational  Depth  Pattern 

Null  hypothesis  1.  There  is  no  significant  difference  in  navigational  depth  patterns  among  left  dominant 
information  processors,  right  dominant  information  processors,  and  integrated  information  processors  when 
navigating  in  a hypermedia  environment. 

Null  hypothesis  1 was  rejected.  There  was  significant  mean  difference  in  the  average  of  navigational  depth 
among  left  dominant  information  processors,  right  dominant  information  processors,  and  integrated  information 
processors  (F  (2 , 99)  = 4.052  , p <.05). 


Table  5.  One-way  ANOVA  for  Navigational  Depth 


Source 

df 

ss 

MS 

F 

I> 

Between 

2 

.0018 

.00089 

4.052  * 

.020 

Within 

99 

.0022 

.00022 

- 

- 

Total 

101 

.0239 

- 

- 

-- 

Post  hoc  Scheffe  analysis  indicated  that  the  comparison  between  navigational  depths  for  left  dominant 
information  processors  and  for  the  right  dominant  information  processor  was  significant  at  the  .05  level  of 
significance. 
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Table  6.  Post  Hoc  Scheffe  Analysis  for  Navigational  Depth 


Comparison  of  Navigational  Depth 

Mean  Difference 

Left  versus  Right 

.0091* 

Left  versus  Integrated 

.0004 

Right  versus  Integrated 

.0087 

*.  The  mean  difference  is  significant  at  the  .05  level. 


Information  Processing  Style  and  Navigational  Path  Pattern 

Null  hypothesis  2.  There  is  no  significant  difference  in  navigational  path  patterns  among  left  dominant 
information  processors,  right  dominant  information  processors,  and  integrated  information  processors  when 
navigating  in  a hypermedia  environment  in  terms  of  the  degree  of  linearity. 

Null  hypothesis  2 was  rejected.  There  was  a significant  mean  difference  in  the  average  of  navigational  path 
among  left  dominant  information  processors,  right  dominant  information  processors,  and  integrated  information 
processors  in  terms  of  the  degree  of  linearity  (F  (2  ,99  ) = 4.089,  p < .05). 

Table  7.  One-way  AN  OVA  for  Navigational  Path 


Source 

df 

SS 

MS 

F 

R 

Between 

2 

.103 

.0513 

4.089 

.020 

Within 

99 

1.241 

.0125 

— 

— 

Total 

101 

1.344 

-- 

- 

- 

Post  hoc  Scheffe  analysis  revealed  that  the  only  comparison  between  navigational  paths  that  was  significant 
at  the  .05  level  of  significance  was  the  comparison  between  navigational  paths  for  right  dominant  information 
processors  and  for  integrated  information  processors. 


Table  8 . Post  Hoc  Scheffe  Analysis  for  Navigational  Path 


Comparison  of  Navigational  Path 

Mean  Difference 

Left  versus  Right 

.0548 

Left  versus  Integrated 

.0220 

Right  versus  Integrated 

.0751* 

*.  The  mean  difference  is  significant  at  the  .05  level. 


Information  Processing  Style  and  Navigational  Method  Pattern 

Null  hypothesis  3.  There  is  no  significant  difference  in  navigational  method  patterns  among  left  dominant 
information  processors,  right  dominant  information  processors,  and  integrated  information  processors  when 
navigating  in  a hypermedia  environment  in  terms  of  navigational  methods  they  employ. 

Null  hypothesis  3 was  rejected.  There  was  significant  mean  difference  in  the  average  of  navigational 
methods  among  left  dominant  information  processors,  right  dominant  information  processors,  and  integrated 
information  processors  in  terms  of  navigational  methods  they  employ  (F  (2  ,99  ) = 5.075  , p <.05). 

Table  9.  One-way  AN  OVA  for  Navigational  Methods 


Source 

df 

SS 

MS 

F 

R 

Between 

2 

.368 

.1840 

5.075 

.008 

Within 

99 

3.589 

.0363 

— 

— 

Total 

101 

3.957 

-- 

-- 

- 

Post  hoc  Scheffe  analysis  revealed  that  the  comparison  between  navigational  methods  for  left  dominant 
information  processors  and  for  the  right  dominant  information  processor  was  significant  at  the  .05  level  of 
significance.  Post  hoc  Scheffe  analysis  also  showed  that  the  comparisons  between  navigational  methods  for  left 
dominant  information  processors  and  for  integrated  information  processors  was  significant  at  the  .05  level  of 
significance. 


ERjt 


149 


143 


Table  10.  Post  Hoc  Scheffe  Analysis  for  Navigational  Methods 


o 

ERLC 


Comparison  of  Navigational  Methods 

Mean  Difference 

Left  versus  Right 

.1296+ 

Left  versus  Integrated 

.1251+ 

Right  versus  Integrated 

.0046 

*.  The  mean  difference  is  significant  at  the  ,05  level. 


Discussion 

The  difference  in  the  average  of  navigational  depth  among  the  three  information  processor  groups  seems  to 
support  the  suggestion  that  right  brain  dominant  subjects  primarily  approach  problem  solving  tasks  in  an  exploratory 
manner,  while  left  brain  dominant  subjects  approach  problem  solving  tasks  in  predominantly  systematic  ways. 

Right  dominant  information  processing  subjects,  as  expected,  accessed  significantly  more  new  nodes  than 
left  dominant  information  processing  subjects.  This  result  is  consistent  with  a similar  study  on  cerebral 
hemisphericity  and  a HyperCard  program  conducted  by  McClusky  (1993).  McClusky  reported  that  right  brain 
dominant  subjects  outperformed  the  other  groups  (left  brain  dominant  subjects  and  integrated  subjects)  in  the  number 
of  new  nodes  visited  within  a HyperCard  program. 

On  the  basis  of  the  findings  of  prior  research  in  the  field  of  hemispheric  information  processing  preference 
and  navigational  patterns,  the  researcher  expected  that  right  dominant  information  processing  subjects  would  prefer 
to  interact  with  hypermedia  systems  in  a relatively  exploratory  and  comprehensive  manner,  visiting  a large  number  of 
new  nodes.  In  addition,  surveying  the  literature  also  led  the  researcher  to  predict  that  left  dominant  information 
processing  subjects  would  prefer  to  interact  with  the  system  in  a systematic  and  conservative  manner,  accessing  only 
very  limited  nodes.  The  result  that  right  dominant  information  processing  subjects  accessed  significantly  more  new 
nodes  than  did  left  dominant  information  processing  subjects  supports  the  researcher’s  expectations  in  conducting 
this  study. 

Given  these  results,  it  might  be  possible  that  right  dominant  information  processing  subjects  explored  the 
hypermedia  system  by  actually  accessing  many  different  nodes  beyond  the  nodes  which  were  necessary  to  visit  to 
perform  the  search  tasks,  while  left  dominant  information  processing  subjects  interacted  with  the  system  by  accessing 
only  the  few  nodes  which  were  required  to  accomplish  the  search  tasks.  It  is  also  likely  that  right  dominant 
information  processing  subjects  accessed  a large  number  of  new  nodes  freely  because  they  would  be  less  concerned 
about  “getting  lost”  problem  within  the  hypermedia  because  of  their  exploratory  manner  for  problem  solving.  In 
addition,  it  is  similarly  likely  that  left  dominant  information  processing  subjects  accessed  very  limited  nodes  because 
they  would  be  very  concerned  “getting  lost”  problem  within  the  hypermedia  because  of  their  conservative  and 
systematic  tendency  of  problem  solving. 

The  observed  frequencies  indicated  that  integrated  information  processors  visited  a larger  number  of  new 
nodes  than  left  dominant  information  processors,  though  the  difference  between  the  two  groups  were  not  statistically 
significant.  One  possible  reason  the  study  failed  to  show  a significant  difference  may  be  due  to  the  relatively  small 
sample  size.  This  should  be  examined  in  future  studies. 

Although  this  result  does  not  provide  statistically  significant  support  for  the  researcher’s  prediction,  it  still 
supports  the  fact  that  there  is  a difference  in  the  navigational  behavior  of  left  dominant  information  processors  and 
integrated  information  processors. 

Other  research  on  hemispheric  preference  for  problem  solving  suggested  that  integrated  information 
processors  are  individuals  who  simultaneously  use  the  left  and  right  dominant  strategies  in  proportion  appropriate  for 
the  situation.  The  results  of  this  study  might  reflect  that  integrated  information  processors  actually  employed  the  left 
and  right  dominant  strategies  at  the  same  time  in  proportion  appropriate  for  the  search  tasks,  while  left  dominant 
information  processors  employed  predominantly  the  left  dominant  strategies. 

The  two  types  of  search  tasks,  open-ended  and  closed-ended  tasks,  were  given  to  the  subjects  to  accomplish 
while  they  interacted  within  the  system.  The  two  different  types  of  tasks  were  given  in  order  to  assist  the  subjects  in 
using  the  hypermedia  program  purposely  and  comprehensively.  The  two  different  types  of  tasks  were  also  given  to 
keep  the  overall  balance  between  the  right  dominant  strategies  and  the  left  dominant  strategies  that  would  be 
demanded  for  accomplishing  the  tasks  in  the  study.  The  open-ended  tasks  used  in  this  research  required  users  to 
respond  in  general  terms  based  on  their  understanding  of  the  overall  content  of  the  hypermedia  system.  Closed- 
ended  tasks  in  this  research  required  users  to  respond  in  specific  terms  that  could  be  found  on  a certain  screen  in  the 
hypermedia  system.  It  was  assumed  that  the  open-ended  tasks  might  require  more  of  the  right  brain  oriented  problem 
solving  strategies,  while  the  closed-ended  tasks  might  demand  more  of  the  left  brain  oriented  problem  solving 
strategies. 

The  results  that  integrated  information  processors  visited  a larger  number  of  new  nodes  than  left 
information  processors  might  lead  the  researcher  to  infer  that  integrated  information  processors  most  likely 
employed  the  left  and  right  brain  dominant  strategies  in  proportion  appropriate  for  the  two  different  types  of  search 
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tasks,  while  left  dominant  information  processors  employed  predominantly  the  left  brain  dominant  strategies  for  the 
both  types  of  search  tasks. 

The  results  of  navigational  path  patterns,  however,  were  contrary  to  the  researcher’s  expectation.  Right 
dominant  information  processing  subjects  followed  significantly  more  linear  paths  than  integrated  information 
processing  subjects.  Right  dominant  information  processing  subjects  followed  a larger  number  of  linear  paths  than 
left  information  processing  style  subjects,  although  not  significantly. 

The  researcher  expected  that  left  dominant  information  processing  subjects  would  prefer  to  navigate 
through  the  system  using  much  more  linear  paths,  while  right  dominant  information  processing  subjects  would  prefer 
to  interact  with  the  system  using  relatively  nonlinear  paths.  This  assumption  was  based  on  information  processing 
style  research  suggesting  that  left  brain  dominant  processors  are  likely  to  approach  problem  solving  tasks 
sequentially,  while  right  brain  dominant  processors  are  likely  to  adapt  different  problem  solving  strategies  such  that 
they  are  likely  to  approach  tasks  non-sequentially. 

It  is  difficult  to  interpret  why  the  results  of  navigational  path  patterns  of  left  and  right  dominant  information 
processors  were  contrary  to  expectation  in  the  current  study.  It  might  be  possible  that  the  contrary  results  were 
found  for  the  navigational  path  patterns  because  the  overall  links  needed  to  accomplish  the  search  tasks  were 
predominantly  linear  themselves.  As  a consequence,  right  dominant  information  processors  who  visited  substantially 
more  nodes  for  the  search  tasks  in  the  study  obtained  higher  scores  in  linearity  value  than  did  left  information 
processors  who  accessed  a lot  less  nodes  within  the  hypermedia  system.  It  is  possible  that  right  dominant 
information  processing  subjects  would  have  outperformed  left  dominant  information  processing  subjects  in  making 
linear  links  if  the  researcher  had  taken  into  consideration  this  aspect  of  A.g.i.l.e.  Resource™  ‘s  operation. 

The  difference  in  the  average  of  navigational  methods  among  three  information  processor  groups  supports 
the  researcher’s  expectations  in  conducting  this  experiment.  On  the  basis  of  the  findings  of  research  on  brain 
hemisphericity  and  navigational  patterns,  the  researcher  assumed  that  left  dominant  information  processing  subjects 
would  prefer  to  navigate  within  hypermedia  systems  using  more  analytical  navigational  methods  such  as  using  index 
and  search  functions.  In  contrast,  the  researcher  expected  that  right  dominant  information  processing  subjects  would 
navigate  hypermedia  systems  using  more  holistic  navigational  features  such  as  maps,  hypertexts,  and  tables  of 
contents. 

The  result  that  left  dominant  information  processors  employed  a significantly  larger  number  of  analytical 
navigational  methods  than  right  dominant  information  processors  confirmed  the  expectation  of  significant 
differences  in  the  navigational  methods  employed  among  the  three  different  information  processor  groups.  The 
results  are  in  agreement  with  the  theory  of  information  processing  styles  supporting  that  left  brain  dominant 
processors  take  predominantly  analytical  approaches  to  problem  solving,  while  right  brain  dominant  processors 
employ  more  likely  holistic  approaches  for  problem  solving. 

The  findings  for  navigational  methods  in  this  study  are  similar  to  the  findings  of  Leader  & Klein  (1996). 
Leader  and  Klein  found  that  the  field  independent  subjects  [who  are  considered  left  dominant  information 
processors]  performed  significantly  better  than  the  field  dependent  subjects  [who  are  considered  right  dominant 
information  processors]  in  using  analytical  navigational  methods  such  as  index  and  find  functions  in  the  experimental 
system.  Indeed,  left  dominant  information  processing  subjects  in  the  present  study  also  performed  the  search  tasks 
employing  substantially  more  of  analytical  navigational  methods  than  right  dominant  information  processors. 

In  addition,  the  findings  of  the  present  study  indicated  that  integrated  information  processing  subjects,  as 
expected,  used  significantly  more  analytical  methods  than  right  dominant  information  processing  subjects.  This 
result  also  is  in  agreement  with  the  findings  of  previous  information  processing  research  for  integrated  information 
processors.  Information  processing  research  suggested  that  integrated  information  processors  are  individuals  who 
use  simultaneously  the  left  and  right  dominant  strategies  in  proportion  appropriate  for  the  situation.  The  result  of 
this  study  might  reflect  that  integrated  information  processors  employed  the  left  and  right  dominant  strategies  at  the 
same  time  while  interacting  within  the  system,  while  right  dominant  information  processors  employed  predominantly 
right  dominant  strategies.  The  researcher  assumed,  based  on  the  theory  of  information  processing  that  integrated 
information  processors  would  employ  analytical  methods  and/or  holistic  methods  in  proportion  appropriate  in  the 
situation  for  the  two  types  of  search  tasks,  while  right  information  processors  would  employ  predominantly  holistic 
methods  for  both  search  tasks. 

Conclusions 

The  results  of  the  study  provide  insight  into  the  relationships  between  navigational  patterns  and  information 
processing  styles  of  hypermedia  users.  These  results,  in  turn,  provide  useful  information  for  advising  learners 
regarding  selecting  more  efficient  and  effective  navigational  patterns,  based  on  their  information  processing  styles. 

In  addition,  the  detailed  information  of  users’  navigational  behaviors  provides  further  guidance  for  designing 
hypermedia  interfaces  that  would  be  more  user-centered.  The  results  of  this  study  also  verifies  the  effectiveness  of 
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hypermedia  systems  in  terms  of  addressing  individual  differences.  The  findings  of  the  present  study  led  to  the 
following  general  conclusions: 

1.  The  present  study  demonstrates  that  the  information  processing  styles  seem  to  play  an  important  role  in  how  an  individual 
interacts  with  hypermedia  systems.  The  three  different  information  processor  groups  used  the  hypermedia  system  in 
significantly  different  ways  in  terms  of  navigational  behaviors.  Left  dominant  information  processors  interacted  with  the  system 
by  visiting  very  limited  nodes  and  employing  a large  number  of  analytical  navigational  methods,  while  right  dominant 
information  processors  interacted  with  the  system  by  accessing  many  new  nodes  and  using  a large  number  of  holistic 
navigational  methods.  The  navigational  patterns  of  integrated  information  processors  also  confirmed  the  assumption  that  this 
group  of  users  used  simultaneously  the  left  and  right  dominant  strategies  in  proportions  appropriate  for  the  situation. 

It  appears  that  the  results  of  the  present  study  provide  valuable  information  for  helping  users  navigate  through  hypermedia 
systems  more  efficiently.  The  results  suggest  that  the  information  processing  styles  of  hypermedia  users  can  be  translated  into  a 
practical  guide  on  advising  users  in  selecting  more  efficient  and  effective  navigational  strategies. 

If  trainers  know  a trainee's  information  processing  style  prior  to  a training  session  or  other  instruction  using  a hypermedia 
system,  they  would  be  able  to  guide  the  different  groups  of  information  processors  in  using  more  efficient  and  effective 
navigational  strategies  for  navigational  depth,  paths,  and  methods.  Also,  when  an  instructional  task  interacting  with  a 
hypermedia  system  seems  to  demand  specifically  left  or  right  brain  dominant  information  processing  strategies,  trainers  or 
teachers  should  advise  right  dominant  information  processors  to  utilize  left  brain  dominant  strategies. 

2.  The  indication  of  the  significant  relationships  between  hypermedia  users’  information  processing  styles  and  their 
navigational  patterns  in  the  present  study  would  also  provide  further  guidance  for  designing  hypermedia  interfaces  which  would 
be  more  user-centered. 

According  to  Sonnier  (1984,  1985,  1989,  1992),  hemispheric  information  processing  preference  is  the  basis  for  the 
common  thread  of  individual  difference  in  the  ways  in  which  human  beings  process  information.  In  addition,  in  a variety  of 
settings,  it  was  verified  that  learners  with  different  information  processing  styles  could  learn  more  and  be  better  motivated  if 
activities  involving  information  retrieval  accommodated  the  information  processing  approaches  of  those  learners  (Cafferty, 
1980;  Carbo,  1980;  Douglass,  1979;  Leader  and  Klein,  1996;  McCluskey,1993).  One  of  the  goals  in  hypermedia  interface 
design,  then,  should  be  to  develop  interfaces  that  are  easy  for  a variety  of  users  with  different  information  processing  styles  to 
navigate  the  systems  by  applying  the  information  processing  strategies  of  those  users.  How  should  hypermedia  designers  design 
such  an  interface? 

The  present  study  could  provide  some  tentative  conclusions  to  guide  more  user-centered  hypermedia  interface  design  that 
would  be  suitable  for  different  user  groups  by  supporting  the  variety  of  information  processing  styles  of  those  users.  In  general, 
hypermedia  designers  and  developers  should  take  note  of  information  processing  style  research  when  making  design  decisions 
related  to  navigation. 

Any  decisions  about  which  navigational  methods  to  include  in  a hypermedia  system  should  take  into  account  the  varying 
types  of  navigational  strategies  used  by  the  three  information  processors.  It  is  worth  noting  that  left  dominant  information 
processors  employed  significantly  more  analytical  navigational  methods,  while  right  dominant  information  processors  used 
more  holistic  navigational  methods.  This  discovery  has  an  important  implication  for  future  design:  the  numbers  of  analytical 
and  holistic  navigational  methods  in  a hypermedia  system  should  be  balanced,  to  a certain  extent.  Or,  the  unbalanced 
implementation  of  navigational  methods  between  analytical  methods  and  holistic  methods,  to  a great  extent,  will  benefit  one  of 
the  information  processing  groups  and  exclude  the  other  when  utilizing  hypermedia  systems.  Leader  and  Klein  (1996)  also 
suggested  that  interface  designers  should  consider  users'  cognitive  style  factors  in  the  use  of  hypermedia  when  they  make 
decisions  for  the  formulation  of  criteria  for  navigational  methods. 

3.  The  results  of  this  study  also  provided  additional  information  that  is  valuable  to  verify  the  effectiveness  of  hypermedia 
systems  in  terms  of  addressing  individual  differences  in  navigational  strategies.  Considerable  educational  research  has 
demonstrated  that  the  quality  of  learning  can  be  enhanced  when  an  instruction  is  matched  to  individual  learning  style  (Carbo  et 
al.,  1986;  Wilson,  1988;  Barba  and  Armstrong,  1992;  Billings  and  Cobb,  1992;  Ellis  et  al.,  1993;  Larson,  1992;  Lee,  1992;  Liu, 
1992;  Overbaugh,  1992).  Hypermedia  has  been  proposed  as  one  of  the  superior  systems  for  achieving  the  goal  of 
accommodating  individual  differences  in  learning  style.  The  underlying  assumption  of  nonlinear  hypermedia  systems  is  that 
different  individuals  will  actively  interact  with  the  systems  and  the  systems  will  accommodate  individual  differences  in  terms  of 
various  users’  navigational  preferences  based  on  ones’  learning  styles. 

The  results  of  this  study  support  that  hypermedia  systems  are  effective  learning  tools  to  address  individual  differences  on 
information  processing  styles  of  hypermedia  users  when  navigating  hypermedia  systems.  The  study  results  indicate  that  the 
users  were  able  to  exploit  the  hypermedia  system  taking  the  navigational  advantages  of  nonlinear  structure  provided  by  the 
hypermedia  system  according  to  their  preferences  of  information  processing  styles. 

In  conclusion,  the  present  study  intended  to  discover  the  effect  of  information  processing  styles  on  the 
navigational  behaviors  of  hypermedia  users.  The  present  study  demonstrates  that  information  processing  styles  seem 
to  play  an  important  role  in  how  an  individual  interacts  with  the  hypermedia  system.  It  is  hoped  that  this  study 
contributes  to  the  body  of  research  on  human-computer  interaction  in  general  and  on  navigational  behaviors  of 
hypermedia  users  in  particular.  The  study  provides  some  research-based  guidance  in  designing  future  hypermedia 
applications  and  indicates  direction  for  future  research  on  navigational  behaviors  of  hypermedia  users.  The 
conclusions  of  this  study,  however,  should  be  considered  suggestions  not  absolute  prescriptions  for  designers  or 
future  researchers  of  hypermedia  systems.  The  conclusions  should  be  confirmed  by  additional  research. 

Hypermedia  systems,  used  as  sophisticated  databases  and  instructional  delivery  systems,  are  becoming  more 
powerful  and  hence  more  popular  in  educational  and  corporate  environments.  The  current  growth  in  the  use  of 
hypermedia  both  in  education  and  corporate  environment  suggests  that  hypermedia  system  will  most  likely  have  an 
extensive  impact  on  our  lives.  As  the  impact  of  hypermedia  on  our  lives  increases,  educational  technologists  should 
continue  research  on  how  the  various  individuals  actually  utilize  hypermedia  and  assist  facilitation  the  use  of 
hypermedia  systems  for  all  types  of  users  in  various  environments  of  our  lives. 
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Recommendations  for  Future  Research 

Further  research  is  definitely  needed  for  a comprehensive  understanding  of  the  issue  of  the  different  users 
and  their  specific  navigational  patterns  of  hypermedia  systems.  The  following  is  a list  of  some  possible  directions 
for  future  research  on  this  topic. 

1 . Future  research  should  include  a replication  of  this  study  across  other  age  groups  to  see  whether  the  results  vary 
significantly  from  this  study.  For  example,  compare  the  differences  between  children  and  adult  learners,  or  elementary  school 
children  and  secondary  students  to  see  whether  navigational  patterns  are  different  when  comparing  one  age  group  to  the  other. 

2.  Future  research  should  also  include  a replication  of  this  study  across  other  groups  of  subjects  from  different  levels  of 

familiarity  with  the  content  of  the  hypermedia  system.  Agility,  to  see  whether  the  results  vary  significantly  from  this  study.  For 
example,  to  see  whether  subjects  who  are  already  somewhat  knowledgeable  in  the  area  of  agility  navigate  differently  than  those 
who  are  not. 

3.  Future  research  will  also  need  to  compare  experienced  and  inexperienced  users  of  hypermedia  systems.  The  experienced 

users  might  have  developed  preferred  patterns  of  navigation,  while  the  inexperienced  users  have  not.  Then,  there  might  be  a 
difference  in  the  way  that  these  two  different  groups  navigate  through  the  material.  In  the  present  study,  most  of  the  subjects 
had  used  a computer  through  computer  seminars,  computer  courses,  or  personal  exploration  with  a computer  in  general,  but 
only  a few  subjects  were  familiar  with  hypermedia  systems.  According  to  Barfield,  Rosenberg,  and  Levasseur  (1991),  there 
exists  a great  deal  of  differences  in  users'  interactions  when  subjects  vary  to  a great  extent  in  their  computer  experience. 

4.  In  addition,  future  research  should  compare  subjects  with  different  learning  demands.  For  example,  see  whether  subjects 
who  need  to  learn  the  material  for  a given  test  navigate  differently  from  those  who  are  simply  evaluating  the  software. 

5.  Future  studies  should  also  investigate  other  hypermedia  applications  to  see  whether  the  results  vary  significantly  from  this 
study.  It  would  be  interesting  to  note  differences  between  and  among  a variety  of  hypermedia  programs  using  identical  groups 
of  subjects. 

6.  In  future  studies,  that  provide  computer  tasks,  more  tasks  should  be  provided  within  each  task  unit.  The  present  study 
only  provided  three  open-ended  tasks  and  five  closed-ended  tasks.  The  limited  numbers  of  tasks  were  given  because  of  the  time 
constraints  in  the  study. 
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Abstract 

Computer  -mediated  communication  technology  provides  a potential  tool  for  facilitating  an  integrated 
science  and  mathematics  curriculum  consonant  with  current  standards.  This  paper  presents  the 
sociocultural  theories  of  L S.  Vygotsky  as  a conceptual  framework  for  implementing  instruction  that 
supports  concept  development  and  promotes  higher  level  thinking  skills  in  students. 


Introduction 

Mathematics  is  the  language  of  science  (Steen,  1994),  and  as  such,  mathematical  understanding  becomes  a 
requisite  for  scientific  literacy.  The  American  Association  for  the  Advancement  of  Science  (AAAS)  (1989)  has 
identified  the  scientific  endeavor  or  the  scientific  enterprise  as  a composite  of  the  dynamic  interrelationship  of 
science,  mathematics  and  technology.  Each  component  involved  in  the  scientific  endeavor  has  its  own  character  and 
history,  yet  each  share  relevant  similarities,  and  each  is  dependent  upon  and  reinforces  the  others.  It  is  the 
similarities  and  interdependencies  among  the  components  of  the  scientific  endeavor  that  suggest  the  efficacy  of  a 
blended  or  an  integrated  curriculum. 

By  extending  the  application  of  Vygotsky’s  theories  to  a technology  facilitated  model  of  an  integrated 
mathematics  and  science  curriculum,  this  paper  will  build  on  a previous  work  that  examined  the  application  of  the 
sociocultural  theories  of  L.S.  Vygotsky  in  the  implementation  of  technology  in  the  mathematics  curriculum  (Harvey, 
& Chamitski,  1998).  In  this  paper  we  will  first  consider  three  major  components  of  Vygotsky's  socio-cultural- 
historical  theories  of  learning,  and  then  discuss  the  relationship  of  these  components  to  the  commonalties  evidenced 
in  the  mathematics  and  science  standards  that  suggest  the  efficacy  of  integrated  curricular  activity.  Next  the 
characteristics  of  computer-mediated  technology  (CMC)  will  be  presented  and  the  rationale  for  its  use  in  an 
integrated  mathematics  and  science  curriculum  will  be  discussed.  Finally,  we  will  outline  a current  project  that  uses 
CMC  as  a facilitating  technology  for  an  integrated  mathematics  and  science  curriculum  that  is  consonant  with  both  a 
Vygotskian  approach  to  learning  and  the  mathematics  and  science  standards. 

Vygotsky's  Theories  of  Sociocultural  Learning 

On  a continuum  of  learning  theory,  Vygotsky's  socio-cultural-historical  theory  lies  between  the  poles  of 
formalism  and  constructivism  (Kozulin,  1990;  Moll,  1990;  Vygotsky,  1986;  Wertsch,  1985).  According  to  Vygotsky 
(1978,  1986,1997;  Vygotsky  & Luria,  1993)  concept  formation  is  an  ongoing  interaction  between  the  concrete  and 
the  abstract  dimensions  that  (contrary  to  constructivism)  does  not  require  that  the  child  reinvent  information,  and 
(contrary  to  formalism)  does  not  expect  the  child  to  form  abstract  conceptualizations  without  first  having  engaged  in 
concrete  activities  that  support  the  formation  of  mental  models.  Three  foundational  ideas  of  Vygotsky's  theories  are 
language,  scientific  and  spontaneous  concepts,  and  the  zone  of  proximal  development. 


Language 

Vygotsky  (1986)  contended  that  language  is  an  indispensable  requisite  for  all  intellectual  growth  and  that 
the  attainment  of  real  concepts  is  impossible  without  words.  Concept  formation  is  "...not  an  isolated,  ossified, 
changeless  formation  but  an  active  part  of  the  intellectual  process  constantly  engaged  in  serving  communication, 
understanding  and  problem-solving."  (p.  98) 

He  argued  that  the  language  of  adults  and  others  which  surrounds  the  child  is  a major  factor  in  the  child's 
cultural  and  social  environment.  This  immersion  in  language  influences  the  content  of  the  child's  thoughts  and 
determines  how  the  child  thinks  (i.e.,  the  interpretation  or  framing  of  information).  Vygotsky  did  not  isolate  verbal 
interchanges  with  adults  as  a separate  component  in  the  process  of  a child’s  concept  formation,  but  rather  stressed  the 
essential  interrelationship  between  sensory  material  and  words  as  mutual  components  of  a child's  concept 
development. 

Two  critical  observations  Vygotsky  (1986,)  made  were  that:  (a)  the  role  of  a child’s  speech  is  equally  as 
important  as  the  role  of  action  in  attaining  a solution  to  a problem;  and  (b)  as  situations  demand  more  complex  and 
indirect  actions  in  finding  a solution  to  a problem,  speech  plays  a more  important  role  in  the  solution  process  as  a 
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whole.  Vygotsky  considered  conceptual  thinking  as  arising  from  a need  to  solve  problems,  and  judged  words  to  be 
the  tools  or  the  means  that  direct  mental  operations  and  focus  their  attention  toward  the  solution  of  a problem. 

Spontaneous  and  Scientific  Concepts 

Vygotsky  (1986,  1997)  identified  two  types  of  concepts,  spontaneous  and  scientific , that  he  distinguished 
by  geneses.  According  to  Vygotsky,  spontaneous  concepts  are  those  concepts  formed  by  the  child  through  everyday 
exposure  to  the  cultural  environment  in  which  he  or  she  lives;  as  such,  spontaneous  concepts  are  typically 
unsystematic  and  highly  contextualized.  Scientific  concepts,  on  the  other  hand,  are  mediated  through  formal 
instruction,  are  typically  derived  from  the  structured  activity  of  classroom  instruction,  and  are  hierarchically  and 
logically  organized. 

The  essence  of  Vygotsky's  approach  to  instruction  is  found  in  his  contention  that,  while  characteristically 
different,  spontaneous  and  scientific  concepts  are  mutually  supporting  structures.  Spontaneous  concepts  give  body 
and  vitality  to  scientific  concepts  and  scientific  concepts  raise  spontaneous  concepts  to  the  level  of  conscious  and 
deliberate  use.  Vygotsky  (Davydov,  1990;  Ratner,  1991;  Schmittau,  1993,  Vygotsky,  1986,  1997)  maintained  that 
formal  concepts  (scientific  concepts)  learned  in  school  must  be  connected  with  the  child's  understandings 
(spontaneous  concepts)  that  result  form  their  everyday  experiences. 

Zone  of  Proximal  Development 

Contrary  to  tenets  of  traditional  cognitivist  thinking,  Vygotsky  (1978)  argued  that  a child's  developmental 
level  should  lead  instruction.  He  maintained  that  a child's  mental  age  should  be  viewed  as  a starting  point  for 
determining  instruction,  and  that  instruction  should  be  configured  in  a manner  that  facilitates  a child's  movement 
forward  through  his  or  her  zone  of  proximal  development  (ZPD).  Vygotsky  defined  the  ZPD  as  the  area  that  lies 
between  a child’s  actual  mental  age  (i.e.  his  or  her  level  of  independent  functioning)  and  the  level  of  problem  solving 
that  the  child  is  able  to  reach  with  assistance.  Simply  stated,  with  adult  mediation  the  child  can  accomplish  more 
than  he  or  she  would  be  able  to  accomplish  independently. 

The  child’s  movement  through  his  or  her  ZPD  is  supported  by  verbal  communication  and  collaborative 
interchange  with  an  adult  or  with  a more  competent  peer.  As  the  child  becomes  more  capable  of  independent 
thinking,  the  child's  state  of  development  rises.  As  the  child’s  developmental  level  increases,  mediation  can  reach 
toward  a higher  instructional  level,  thus,  instruction  leads  development. 

Movement  through  the  ZPD  involves  intelligent,  conscious  imitation,  which  requires  that  the  child  first 
understand  the  field  structure  and  relationships  between  objects.  Imitative  behavior  that  is  characteristic  of 
movement  through  the  ZPD  should  not  be  confused  with  automatic  imitative  behavior  that  shows  no  signs  of 
conscious  understanding. 

Similarities  Between  Mathematics  and  Science 

Mathematics  and  science  are  intellectual  and  social  undertakings  that  share  the  underlying  ideals  of  order; 
honesty  and  openness  in  reporting;  peer  review  and  outside  validation  of  work.  NAAS  (1989)  stated  that  proficiency 
in  both  the  mathematics  and  science  disciplines  requires  extensive  student  experience  using  relevant  principles  to 
solve  problems,  to  communicate  ideas,  and  to  connect  and  generalize  concepts.  Participants  in  mathematical  and 
scientific  enterprises  require  imagination  as  they  apply  their  human  intelligence  to  figuring  out  the  workings  of  the 
world  and  its  myriad  of  connections. 

The  standards  established  by  the  National  Council  of  Teachers  of  Mathematics  (NCTM,  1989,  1991,1995) 
and  the  National  Research  Council  (NRC,  1996)  have  given  direction  to  the  current  reform  efforts  in  the  area  of 
mathematics  and  science  education.  According  to  Steen  (1994)  the  following  are  the  similarities  between  the 
Standards  for  School  Mathematics  (NCTM,  1989)  and  the  National  Science  Education  Standards  (NRC,  1996): 

■ both  advocate  a significant  shift  away  from  the  "filter"  model  of  education  by  raising  expectations  for  all  student  learning 

■ both  argue  for  more  depth,  understanding,  and  thinking  and  less  memorization,  mechanics,  and  mimicry 

■ both  advocate  active  learning,  increased  collaboration,  discussion,  exploration,  and  "student  talk"  and  less  rote  learning, 
decreased  deference  to  teacher  & text  as  sole  authorities  and  sources  of  validation 

■ greater  reliance  on  performance-based  instruments  that  are  coordinated  with  the  curriculum  and  embedded  in  instruction 
and  viewed  as  a recursive  and  reflective  process 

■ both  advocate  full  use  of  technology  as  a means  and  as  a goal  of  instruction 

Vygotsky’s  theories  imply  a multidimensional,  holistic  approach  to  learning;  the  same  type  of  learning  that 
is  prescribed  by  both  the  mathematics  and  science  standards,  and  required  by  the  prerequisite  skills  for  discipline 
involvement.  Learning  experiences  consistent  with  a socio-cultural-historical  approach  are  steeped  in  meaning- 
making colloquium,  connectivity,  and  multi-layered  or  multi-tasked  instruction  (i.e.,  instruction  that  engages 
participants  in  subtasks  at  different  cognitive  levels,  each  of  which  contribute  to  a larger  common  task).  Both  sets  of 
standards  are  aligned  with  Vygotsky  in  their  imperative  to  position  conceptual  learning  over  memorization  and 
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automatic  mimicry,  situate  content  in  leveled  tasks,  and  engage  students  in  support  collaborative  and  cooperative 
enterprises.  Agreement  also  exists  in  the  proposition  that  the  conceptual  learning  of  the  mathematics  and  science 
disciplines  requires  learners  to  be  actively  engaged  in  contextualized,  authentic,  problem-based  learning  that  is 
facilitated  and  guided  by  the  teacher.  In  this  learning  context,  students  are  characterized  by  their  participation  in 
meaningful  communication  and,  and  interactive  engagement  with  educational  facilitators,  with  each  other,  and  with 
their  environment. 

The  Role  of  Computer-Mediated  Communication 

The  very  nature  of  our  technologically  driven  society  elevates  the  status  of  collaboration  and  connectivity  in 
the  processes  of  gathering,  applying  and  sharing  information  in  the  learning  process.  The  growing  availability  of 
computer  mediated  communications  (CMC)  in  the  school  setting  introduces  new  tools  that  provide  extensive 
opportunity  for  curricular  integration  of  authentic  activities,  engagement  in  outside  collaboration,  access  to  vast 
databases  of  information,  and  efficient  and  economical  means  of  information  dissemination,  sharing,  and  validation. 

The  NSF  issued  a joint  document  with  the  U.  S.  Department  of  Education  (1995)  which  stated  that  the 
appropriate  use  of  technology  has  the  potential  to  improve  teaching  and  learning;  expand  and  enrich  opportunities 
for  learning;  link  schools  and  learning  sites  to  provide  students  access  to  the  broader  society;  and  provide  equal 
access  to  educational  opportunities. 

Computer  Mediated  Communication  (CMC)  is  an  all-inclusive  term  that  embodies  technologies  that 
mediate  communication  between  and  among  individuals  and  groups  of  individuals.  Subsets  of  CMC  would  include 
but  not  be  limited  to  e-mail,  the  World  Wide  Web  (WWW),  listserves,  and  online  searchable  databases. 

Riel  (1996)  asserted  that  computer-mediated  communication  (CMC)  is  a tool  that  can  narrow  the  gap 
between  collaborative  and/or  cooperative  experiences  and  traditional  formal  instruction.  Scott  (as  cited  in  Jonassen, 
1996)  evidenced  that  students  working  in  groups  through  CMC  participated  more  evenly,  and  accomplished  more 
task  objectives  than  students  not  using  CMC. 

The  nature  of  asynchronous  communications  allows  participants  a delayed  time  frame  that  is  generally  not 
afforded  in  the  traditional  classroom  setting.  Romiszowski  (1997)  contended  that  this  window  of  time  may  serve  to 
enhance  reflective  thought  and  encourage  participation.  The  author  maintained  that  CMC  is  “...most  promising  for 
the  development  of  the  reflective  thinking  and  creative  planning  skills  that  are  required  to  close  the  gap  between 
information  and  performance  in  knowledge  work.”  (p.  33) 

Kahn  (1997b)  characterized  CMC  as  an  interactive,  open  system  that  offers  global  accessibility  that  is 
distance  and  time  independent.  He  further  noted  that  CMC  offers  a high  degree  of  learner  control  and  environmental 
flexibility.  CMC  can  support  both  formal  and  informal  learning  environments  (McLellan,  1997),  collaborative 
learning  (Hiltz,  1994),  and  has  the  potential  to  extend  cooperative  and  collaborative  learning  experiences  to  every 
classroom  that  is  connected  to  the  Internet  (Relan,  & Gillani,  1997). 

Shneiderman,  Alvai,  Norman  and  Borkowski  (1995)  stated  that  ” The  low  cost  and  flexibility  of  e-mail  and 
Web  usage  means  these  technologies  are  likely  to  become  universal..."  (p.41)  They  liken  e-mail  to  the  personal 
automobile  as  a means  of  information  transportation;  most  people  will  have  it  and  use  it  daily.  Listserves  and  Web 
sites  they  compare  to  commuter  trains;  large  numbers  of  people  will  use  it  regularly  to  get  to  specific  locations. 

Sabelli  (1995)  stressed  the  need  for  the  incorporation  of  advanced  technology  in  the  classroom 
environment.  She  asserted  that  as  the  affordability  of  powerful  networked  technologies  increases,  educators  must 
assume  responsibility  for  the  profound  pedagogical  implications  of  change  these  technologies  bring  into  the  areas  of 
science,  mathematics  and  engineering.  Sabelli  contended  that  when  technologies  are  properly  used  in  the 
instructional  process,  they  have  the  potential  to  make  science  and  mathematics  more  understandable  and  attractive  to 
an  increasing  number  of  students. 

Computer  technology  has  changed  the  way  society  thinks,  works,  studies  and  plays.  Although  Vygotsky 
died  long  before  our  current  state  of  technological  advancement  was  even  dreamed  of,  he  provided  a theory  in  which 
the  everyday  experiences  (i.e.,  spontaneous  concepts)  of  the  child  weigh  heavily.  Computer  technology  is  a part  of 
the  child's  everyday  experiences,  from  talking  cash  registers  and  digital  scanners  in  grocery  stores  to  video  games 
and  surfing  the  net.  Accordingly,  instruction  should  be  tied  to  these  everyday  experiences. 

The  increasing  access  to  CMC  and  its  collaborative,  flexible  nature  appear  to  make  it  a viable  instructional 
tool  for  the  implementation  of  activities  that  are  consonant  with  both  the  curricular  disposition  and  discourse 
promoted  by  the  mathematics  and  science  communities  and  with  Vygotskian  theory. 

Curriculum  Integration 

While  effective  curriculum  integration  involves  a great  deal  of  time  to  plan  and  implement  (Brant,  1991; 
Dressel,  1958;  Jacobs,  1989;  Palmer,  1991),  the  effects  on  student  learning  appear  to  make  it  well  worth  the  invested 
effort  (Levitan,  1991;  Maclver,  1990).  Lipson,  Valencia,  Wixson,  and  Peters  (1993)  asserted  that  the  positive 
effects  on  students  who  engage  in  appropriately  integrated  curricula  include:  (a)  more  robust  application  of  skills; 

(b)  faster  retrieval  of  information;  and  (c)  increased  depth  and  breadth  of  learning. 
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Palmer  (1991)  listed  the  essential  elements  of  an  integrated  curriculum  as:  (a)  core  skills  and  processes;  (b) 
curriculum  strands  and  themes;  (c)  major  themes;  (e)  questions;  (f)  unit  development;  and  (f)  evaluation. 

Mathematics  and  science  emphasize  the  following  common  skills  and  processes  that  directly  affect  methodology  in 
both  disciplines  (Berlin  & White,  1994;  Davidson,  1995;  NCTM,  1989;  AAAS,  1989): 

■ observing 

■ collecting,  recording,  and  organizing  data 

■ recognizing  time/space  relationships 

■ communicating 

■ interpreting  data 

■ graphing 

■ predicting 

While  the  literature  is  replete  with  examples  of  CMC’s  use,  or  potential  use  in  classrooms  across  the  world 
(e.g.,  Dede,  1996;  Jonassen,  1996;  Goldberg,  & Richards,  1995;  Kahn,  1997a;  Morrison,  & Goldberg,  1996),  we 
will  discuss  the  organization  of  a project  in  which  we  are  currently  involved  as  an  example  of  how  curriculum  and 
methodology  may  be  blended  in  mathematics  and  science. 

The  goal  of  this  project  is  to  have  fifth  grade  middle-school  students  recognize  and  apply  their  existing  and 
emerging  mathematical  repertoire  to  a science  unit  on  weather.  The  students  are  heterogeneously  grouped,  and  the 
project  is  structured  to  allow  students  to  engage  in  collaborative  ventures  that  contribute  to  a cohesive,  communal 
product  from  multi-  leveled  and  topic-divergent  activities.  Evaluation  is  formative  and  based  on  teacher-constructed, 
and  teacher  and  class-constructed  rubrics.  This  project  is  configured  to  incorporate  Palmer's  (1991)  six  essential 
components  of  integrated  curricula. 

In  the  beginning  of  the  project,  students  engage  in  three  concurrent  activities:  (a)  learning  basic  weather 
concepts  such  as  types  of  clouds,  rain,  snow,  fog,  windspeed  through  a variety  of  hands-on  experiments;  (b) 
exploring  diverse  geographical  areas  using  the  World  Wide  Web  which  includes  recording  each  site's  map  location 
(including  longitude  and  latitude),  defining  topographical  features,  seasonal  cycles,  and  identifying  characteristics, 
and  (c)  subscribing  to  relevant  listserves. 

While  learning  about  basic  weather  concepts,  students  explore  the  situational  measurements  of  latitude  and 
longitude.  They  will  then  place  various  locations  according  to  their  latitude  and  longitude  coordinates.  Students  may 
also  relate  latitude  and  longitude  to  things  such  as  seasonal  cycles,  overall  weather  patterns,  and  time  zones. 

( recognizing  time/space  relationships)  After  discussing  and  sharing  the  preliminary  information  collected,  the 
students  come  to  consensus  on  how  this  information  should  be  categorized,  and  hierarchically  organized. 
(communicating;  collecting , recording,  and  organizing  data) 

During  the  next  phase  of  the  project,  each  students  group  will  select  a specific  geographical  area  of  interest 
and  begin  to  collect  previously  agreed  upon  weather  data  (e.g.,  wind  speed,  temperature,  precipitation,  wind-chill, 
heat  index)  and  other  relevant  information  from  one  of  the  many  weather  sites  found  on  the  Internet  (e.g., 

WeatherNet  at  http://cirrus.sprl.umich.edu/wxnet/),  as  well  as  from  newspapers,  periodicals,  radio,  and  television 
(e.g.,  The  Weather  Channel).  Students  will  also  use  other  weather  information  tools  easily  found  on  the  Internet 
such  as  weather-cams,  (e.g.,  http://cimis.sprl.umich.edu/wxnet/wxcam.html),  related  software  (e.g., 
http://cimis.sprl.umich.edu/wxnet/software.html),  computer  forecast  models  (e.g., 
http://cimis.sprl.umich.edu/wxnet/model/model.html),  weather  satellite  transmissions  (e.g., 
http://www.intellicast.com/weather/mem/nexrad/),  and  weather-sites  ( observing;  collecting  and  recording  data; 
interpreting  data). 

Group  data  on  each  geographical  area  will  be  entered  into  databases  and/or  spreadsheets.  Graphs  will  be 
constructed  as  well  as  other  visual  representations  that  will  help  each  group  communicate  their  data  to  the  other 
student  researchers  ( communication ; graphing;  interpreting  data).  When  sufficient  background  information  has 
been  collected  on  the  different  geographical  locations,  the  students  will  bring  their  data  to  the  whole  class  at  which 
time  the  all  of  the  student-researchers  will  discuss,  compare  and  relate  findings  (e.g.,  Buffalo,  New  York  and  Grand 
Rapids,  Michigan  may  be  experiencing  similar  amounts  of  snowfall  due  to  the  lake  effect)  (communication, 
interpreting  data , make  predictions).  During  the  final  phase  of  instruction,  students  will  follow  major  weather  fronts 
and  make  weather  predictions  based  on  their  accumulated  knowledge  and  past  observations. 


Conclusions 


There  appears  to  be  growing  acceptance  by  educators  of  both  the  views  of  curricular  content  and  student 
discourse  promoted  in  the  Mathematics  and  Science  Standards  (Chang-Wells,  & Wells,  1993;  Cobb,  Wood,  & 
Yakel,  1993),  and  of  Vygotsky’s  theories  of  the  socio-cultural  nature  of  learning  (Tharp  & Gallimore,  1988).  The 
American  Psychological  Association  (1995)  stated  that  environmental  factors  such  as  culture,  technology,  and 
instructional  practices  along  with  social  interaction  have  substantial  influence  over  learning.  Bonk  and  Thomas 
(1997)  asserted  that  there  is  increased  agreement  among  educators  that  meaning  is  found  in  the  process  of 
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negotiation  and  knowledge-building  in  a social  context.  Curtis  and  Reynolds  (1997)  stated,  ‘The  learner-centered 
movement  has  encouraged  instructors  to  create  challenging  and  novel  environments  that  help  learners  link  new 
information  to  old,  seek  meaningful  knowledge,  and  think  about  their  own  thinking.”  (p.  167) 

CMC  is  a tool  that  appears  to  hold  significant  promise  for  supporting  teachers’  efforts  in  this  direction.  The 
defining  characteristics  of  CMC  and  its  unique  ability  to  extend  and  to  enhance  semiotic  exchange  make  it  a 
potential  vehicle  for  supporting  integrative  classroom  environments  that  are  consonant  with  the  mathematics  and 
science  standards  and  Vygotskian  theory  relative  to  language,  spontaneous  and  scientific  concepts,  and  the  zone  of 
proximal  development. 
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Abstract 


This  research  assumes  that  systematic  and  systemic  change  are  two  components  in  instructional  design  - 
not  dichotomous  terms ; but  two  sides  of  an  interconnected  dual  structured  system.  Specifically , this 
presentation  tries  to  determine  the  difference  between  systemic  thinking  and  systematic  thinking , the  ideal 
mental  model  (dual  structured  systems)  for  ID,  and  the  benefits  from  this  mental  model. 

Needs  for  Dual  Structured  Instructional  Systems  Approach 

Systems  thinking  has  come  to  the  attention  of  social  scientists,  planners,  operational  engineers,  managers, 
educators,  and  the  like,  as  a tool  for  problem  solving  and  decision  making.  According  to  the  SCANS  (The  Secretary 
of  Labors  Commission  on  Achieving  Necessary  Skills)  report  (1991),  systems  thinking  is  an  important  problem 
solving  skill  needed  both  in  school  and  industrial  training  curricula.  Also,  Reich  (1991)  has  predicted  that  systems 
thinking  will  be  an  essential  skill  in  promising  job  areas  like  symbolic  analyst. 

We  might  easily  assume  that  if  instructional  designers  (ID)  are  to  join  as  members  of  symbolic  analyst 
group,  they  should  also  possess  systems  thinking  skills.  Then  how  can  they  learn  how  to  increase  these  skill?  At 
present,  there  are  no  essential  sources  on  how  to  teach  and  learn  systems  thinking  skills.  There  is  not  even  a clear 
definition  of  or  guidelines  for  system  thinking. 

Historically,  the  instructional  design  field  has  employed  systems  approaches  under  the  name  of  “systematic 
approaches.”  Dick  and  Carey's  “systematic  design  of  instruction”  (1990)  can  be  selected  as  representative  material 
for  these  approaches.  According  to  them,  the  instructional  systems  design  (ISD)  process  is  regarded  as  a procedure 
system,  a series  of  steps  such  as  design,  development  and  implementation.  The  ISD  model  has  recognized  a means 
for  not  only  effectively  carrying  out  large-scale  design  and  development  efforts  but  also  a means  for  ensuring  quality 
control.  The  common  language  established  by  the  model  and  its  procedures  facilitates  communication  among  team 
members  (McCombs,  1986). 

However,  it  is  argued  that  if  instructional  designers  only  followed  the  steps  given,  then  quality  instructional 
productions  can  be  created  in  the  same  manner  as  industry  productions  on  a conveyer  system.  In  this  approach,  most 
of  the  ID  might  be  involved  in  a series  of  activities  such  as  dividing  tasks  into  small  chunks  (reductional  activities), 
finding  step-by-step  sequences  (linear  thinking),  etc.  Thus,  analytic  thinking  skill  can  be  important  for  ID. 

On  the  contrary,  there  is  another  group  of  IDs  (Banathy,  Reigeluth,  etc.)  armored  with  the  systemic  point  of 
view.  They  are  regarded  as  a group  which  seeks  authentic  systems  approaches.  They  use  more  the  broad  term  like 
educational  design  instead  of  the  narrow  term  like  instructional  design.  The  systemic  approach  provides  instructional 
design  practitioners  with  “the  dynamics  of  process.”  for  dealing  with  complex  real-world  instructional  problems. 

This  approach  suggests  that  the  design  of  education  should  be  more  open  and  adaptive  to  the  larger  societal  system. 

In  this  approach,  ID  has  to  develop  a holistic  viewpoint  or  synthetic  thinking  skills. 

In  this  context,  we  think  both  the  systemic  and  the  systematic  approach  are  important  and  necessary  in  ID. 
The  systemic  approach  provides  a conceptually  sound  framework,  which  ensures  the  understanding  of  the  complex 
and  dynamic  relationship  between  instructional  system  and  its  environment.  The  systematic  approach  provides 
actionable  solutions,  which  are  easy  to  follow.  Thus  we  assume  that  systematic  and  systemic  are  two  components 
interdependent  with  each  other  in  the  instructional  systems  design.  They  are  not  dichotomous  terms;  rather,  the  two 
sides  are  an  interconnected  dual  structured  system. 

Specifically,  this  article  tries  to  determine: 


First,  what  is  the  difference  between  systemic  thinking  and  systematic  thinking? 
Second,  what  might  be  the  ideal  mental  model  (dual  structured  systems)  for  ID? 
Third,  what  benefits  can  we  find  from  the  mental  model? 
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Overview  of  the  problems  relating  to  systematic  and  systemic 
Dictionary  definition 

According  to  several  dictionaries  (The  Oxford  Dictionary,  1989;  The  Random  House  Dictionary,  1983; 
Webster’s  Dictionary,  1981),  The  meanings  of  systematic  and  systemic  are  as  follows: 

A.  Systematic: 

(1 ) Orderly  scheme  or  plan 

(2)  Classification,  procedure 

(3)  Manifesting  or  involving  a system 

B.  Systemic: 

(1)  Body  as  a whole 

(2)  Distinguished  from  local 

(3)  Pertaining  to  or  affecting  a particular  system 

In  terms  of  the  first  definition,  systematic  seems  to  be  an  orderly  scheme  or  plan  and  systemic  seems  to  be 
body  as  a whole.  However,  from  the  third  definition,  we  learn  that  the  terms,  systematic  and  systemic  both  have  a 
relationship  with  the  term  system.  Let  us  say,  the  two  terms  come  from  the  same  root,  system.  Sometimes  they  are 
difficult  to  separate  and  even  interchangeable.  Systematic  seems  to  be  more  a generic  term,  more  well-known  to 
non-experts  than  systemic. 

The  different  approaches  of  the  two  terms  in  the  field  of  ISD 

A.  Systematic  design  of  instruction  (Dick  and  Carey,  1990) 

Dick  and  Carey  (p.  2-11)  generally  discuss  the  systematic  approach  model  for  designing  instruction.  They  begin  with  an 
explanation  of  the  relationship  between  a heating/cooling  system  (a  sort  of  hard  system)  and  an  instructional  process  in 
terms  of  systems  concept.  They  insist  that  the  purpose  of  their  work  is  to  describe  a systems  approach  model  for  the 
design,  development,  implementation,  and  evaluation  of  instruction.  They  define  their  approach  as  a procedural  system,  a 
series  of  steps,  each  of  which  receives  input  from  the  preceding  steps  and  provides  output  for  the  next  steps. 

B.  System  design  of  education  (Banathy,  1991) 

Banathy  talks  about  the  importance  of  the  systemic  viewpoint  as  a more  fundamental  approach  to  solving  problems  in 
educational  situations.  As  an  important  aspect  of  the  systemic  approach,  he  stresses  that  the  design  of  education  should  be 
more  open  and  adaptive  to  the  larger  societal  system.  He  is  explicit  in  his  criticism  of  the  systematic  approach  without 
considering  systemic  aspect  in  educational  reform. 

The  ship  of  education  is  sailing  on  troubled  waters.  There  is  an  ever-increasing  realization  that  unless  we 
change  the  course,  the  ship  will  sink.  But  people  are  still  trying  to  “rearrange  the  chairs”  (systematic)  on  the  deck  of 
the  sinking  ship(p.6). 

Distinction  based  on  Hard  systems  Vs.  Soft  systems 

We  also  assume  that  there  might  be  some  typical  tasks  (hard  systems,  step-by-step  procedures,  etc.)  related 
with  the  systematic  approach,  some  typical  events  (heuristic  problems,  soft  systems,  etc.)  related  with  the  systemic 
approach  and  other  combinational  events  needing  both  approaches  (most  of  the  tasks  in  real  situations  might  be  these 
cases). 


A.  Systematic  approach  rooted  in  hard  systems  perspective 

Systematic  approaches  from  systems  engineering  were  translated  into  instructional  systems  design  for  military  application 
(Settler,  1990).  As  a result, , hard  systems  thinking  prevails  overwhelmingly  in  ID  practice. 

Saettler  (1990)  summarizes  how  a systems  approach  based  on  the  hard  tradition  is  practical  in  the  instructional  design 
process: 


Instructional  goals  and  objectives  were  precisely  defined,  various  alternative  were  analyzed, 
instructional  resources  were  identified  and  /or  developed,  a plan  of  action  was  devised,  and  results  were 
continuously  evaluated  for  possible  modification  of  the  program  (p.  350). 

His  statement  implies  that  an  instructional  outcome  can  be  achieved  through  “a  series  of  steps  activities” 
(systematic  approach)  carried  out  in  some  predetermined  sequence. 

B.  Systemic  approach  rooted  in  soft  system  perspective 

The  systemic  ID  process  embedded  in  soft  systems  thinking  must  go  through  an  iterative  cycle  as  new  experiences  and 
insight,  such  as  knowledge  resulting  from  one  design  episode  are  gained.  Through  this  iterative  design  process, 
relationships  among  the  systems  that  surround  an  instructional  enterprise  will  be  better  understood  and  instructional 
solutions  refined.  Therefore,  the  design  process  will  become  as  ongoing  learning  process  in  flux. 
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Systemic  thinking  provides  ID  with  “the  dynamics  of  process”  for  dealing  with  complex  real-world 
instructional  problems.  Instructional  development  grounded  in  soft  systems  thinking  aims  to  be  practiced  as  a social 
process.  Thus,  this  approach  represents  a collaborative  act  where  stakeholders  engage  in  an  interpretive 
understanding  of  instructional  development  through  dialogue.  It  thus  seeks  accommodation  among  conflicting 
interests  (Checkland,  1985,  p.  764). 

Instructional  development  is  a part  of  larger  systems,  such  as  the  school  system,  the  community  system,  or 
the  total  social  system.  Thus,  ID  must  see  the  design  process  as  linked  with  the  instructional  program,  the 
community,  the  human  performance  system  in  which  skills  or  knowledge  gained  from  an  instructional  intervention 
will  be  used,  and  the  surrounding  social  system,  all  concurrently. 

In  most  cases,  the  total  set  of  problems  of  ID  practice  in  educational  and  business  contexts  cannot  be 
understood  clearly.  ID  practitioners  often  face  many  complex  situations  in  which  multiple  actors  (e.g.,  teachers, 
students,  and  subject  matter  experts)  and  goals  interact  intermittently.  This  implies  that  practitioners  in  the  field  of 
instructional  development  need  to  develop  flexible  interventions  capable  of  adapting  to  emerging  objectives,  needs, 
and  problems  (e.g.,  instructional  constraints).  In  so  doing,  the  design  process  becomes  embedded  in  a context,  and 
takes  the  form  of  iterative,  collaborative  inquiry.  It  might  be  necessary  for  ID  to  develop  more  systemic  thinking 
skills. 

The  conceptual  model  of  dual  structured  instructional  systems 

In  our  dual  structured  instructional  systems,  Systemic  and  systematic  are  not  longer  conflicting  approaches 
any  more.  These  two  approaches  provide  complementary  necessary  knowledge  bases.  A more  detailed  conceptual 
model  is  shown  in  Figure  1 . 


Figure  1.  Visual  model  of  dual  structured  instructional  systems 


Providing  conceptually  sound 
framework 


Providing  feasible  solution  and  feedback 
for  elaborating  framework 


Systemic  Approach 

Knowledge  base  for  conceptual  level 
(problem  understanding  and  construction  ) 

Conceptual  (understanding  level) 

Synthetic 

Dynamic 

Multi-layers  (levels) 

Global 

Double  loop  process  (meta-cognitive  strategic) 
More  construed vistic 
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intervention) 
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Some  guidelines  for  increasing  systemic  thinking 

Now,  we  do  not  want  to  give  guidelines  for  systematic  thinking  because  most  of  the  IDs  are  already  familiar 
with  that  skill.  We  want  to  categorize  systemic  thinking  as  several  sub-abilities  and  develop  some  guidelines  to 
increase  systemic  thinking  skill: 
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• Ability  to  see  the  problem  as  holistic. 

The  designer  needs  to  study  systems  theory  as  a disciplines.  The  designer  should  be  familiar  with  visual  data  like  graphics, 
images,  charts,  drawings,  etc.  To  obtain  a holistic,  imagery  and  intuitive  point  of  view,  visual  literacy  might  be  better  than 
verbal  literacy  because  verbal  expression  is  subject  to  logical  and  linear  thinking. 

• AbilittLtCLSet  goals  in  given  problemsolving  situation. 

Given  a problem  situation,  the  designer  should  be  able  to  change  the  problem  into  a goal.  This  ability  can  be  raised  by 
need  analysis  because  the  goal  setting  phase  is  part  of  the  analysis.  Finding  goal  activities  leads  the  designer  to  develop 
his/her  purpose  seeking  sense  in  a broader  area.  Systematic  thinking  also  might  deal  with  goal  finding  activities.  However, 
in  this  case,  the  activities  deal  with  simply  changing  problems  into  goals  without  any  holistic  perspective. 

• Ability  to  link  possible  resources  in  terms  of  goals  in  a given  problem  situation. 

To  do  so,  the  designer  should  be  familiar  with  a multi-disciplinary  approach.  It  might  make  possible  resources  meaningful 
and  useful  for  solving  the  problem.  Only  when  a designer  sees  all  the  resources  available  as  interdependent  in  terms  of  the 
goal,  it  is  possible  that  creative  solutions  can  be  developed. 

• Ability  to  drive  a tentative  result  (changes  in  system  in  terms  of  time  and  space)  in  a given  condition  through  test. 

There  might  be  several  kinds  of  methods  to  make  the  designer  think  systemically. 

1.  One  method  is  to  conduct  the  actual  experiment  in  terms  of  models  (models  for  solving  problems).  When  a project  is 
longitudinal  and  tasks  are  complex,  it  might  be  a very  time  consuming  and  costly  approach. 

2.  Secondly,  using  computer  simulation  might  be  more  cost  efficient  than  conducting  an  actual  experiment.  If  the 
designer  chooses  several  options  among  lots  of  input  values,  the  computer  quickly  presents  items  corresponding  to 
the  selected  option. 

3 Thirdly,  through  the  various  case  studies  containing  real  situation  problems,  the  designer  also  leani  how  to  think 

systemically.  In  this  case,  the  problem  solving  approach  should  be  systemic.  There  might  be  several  guidelines  given 
to  increase  systemic  thinking  ability. 

4.  Fourth,  thought  experiment  might  be  the  most  time -cost  efficient  method  because  human  thought  can  go  anywhere  at 
anytime.  But  it  is  very  difficult  to  get  a valid  result  after  administrating  thought  experiment.  So  follow-up  activities 
through  which  the  designer  can  find  existing  data  from  similar  cases  or  scientific  results  should  be  involved.  Also, 
the  activities  should  be  based  upon  theory  or  models.  Some  practical  tips  can  be  obtained  from  other  fields  like 
physics  (Indeed,  Albert  Einstein  used  this  method). 

The  benefits  of  Dual  Structured  Systems  Approach . 

There  might  be  several  benefits  of  Dual  Structured  Systems  Approach,  including: 

• First  of  all,  it  can  eliminate  confusion  between  two  aspects  (systematic/systemic)  because  this  approach  incorporates 
the  two. 

• This  approach  can  drive  further  study  because  the  mental  model  still  needs  to  be  improved  and  elaborated. 

• There  might  be  some  possibility  that  IDs  can  know  when  and  where  one  aspect  should  be  more  dominant  than  the 

other  during  the  instructional  design  process.  For  example,  If  the  task  is  simple  and  procedural,  then  the  designer 
may  use  systematic  thinking  like  logical  and  step-by-step  analytic  ability.  On  the  contrary,  if  the  task  is  complex  and 
heuristic  like  soft  knowledge,  then  the  designer  may  use  systemic  (synthetic)  thinking;  let  us  say,  the  ability  to  see 
the  problem  as  holistic  and  to  have  a birds-eye  perspective,  which  lead  to  find  out  some  principles  or  relationship 
among  several  factors.  Thus,  in  this  case,  the  designer  might  be  requested  to  handle  a lot  of  factors  needed  to  solve 
the  problem.  Perhaps,  the  factors  might  be  appeared,  at  a glance,  as  unrelated  things  in  the  sight  of  the  systematic 
oriented  designer. 

• Finally,  IDs  might  know  the  meaning  of  “think  globally  (systemic),  act  locally  (systematic).”  Let  us  say,  IDs  could 

gain  the  ability  to  design  in  a more  broader  societal  domain  and  to  apply  more  systematic  methods. 


Conclusion 

HD  TV  which  was  made  in  Japan,  was  a kind  of  innovation  that  has  finally  been  proven  as  a failure 
because  the  developing  group  did  not  think  of  the  systemic  aspect,  let  us  say,  because  environmental  change  (societal 
trends)  go  to  digitized  but  they  did  not  agree  with  that.  So,  if  we  design  an  innovation,  we  have  to  think  about  the 
systematic  approach  based  upon  a systemic  viewpoint  because  it  reduces  the  probability  of  failure. 

As  another  example,  before  we  build  a building,  we  need  to  think  about  the  outside  of  the  system  (building) 
like  the  quality  of  ground,  strength  of  wind,  etc.  So,  it  is  very  important  to  think  about  the  systemic  point  of  view 
before  we  design  a system  to  permit  the  system  to  develop  safely  and  soundly. 

On  the  contrary,  when  we  start  to  do  something  in  terms  of  performance  level,  it  might  be  more  practical 
and  efficient  to  have  a systematic  viewpoint. 

We  need  both  systemic  and  systemic  approaches  for  instructional  design.  Also,  we  have  to  know  when 
or  where  systematic  and  systemic  approaches  are  useful.  So  further  study  is  needed  to  figure  out  these 
aspects. 
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Abstract 

This  study  intended  to  determine  the  effects  of  teaching  metacognitive  strategies  on  performance  in  a self- 
directed  learning  situation.  All  participants ; 60  university  students  enrolled  in  a beginning  photography 
course  for  non-art  majors , were  subject  to  the  same  conditions.  The  treatment  was  embedded  instruction 
and  practice  in  reflection , planning  and  evaluation.  Metacognitive  awareness  was  measured  prior  to  and 
after  the  treatment.  The  use  of  metacognitive  strategies  was  measured  by  a self- reflection  survey,  following 
the  first  and  last  assignment.  These  assignments  were  identical  and  provided  for  the  assessment  of 
performance.  Results  indicated  that  the  treatment  had  a positive  effect  on  learning.  The  change  in 
metacognitive  awareness  led  the  researchers  to  conclude  that  instructional  strategies  which  teach  students 
to  practice  metacognitive  skills  while  learning  course  content  improves  the  use  and  awareness  of  these 
skills  as  well  as  performance. 

Introduction 

Metacognition  is  the  ability  to  reflect,  control  and  understand,  in  a self-aware  mode,  one's  own  learning  and 
cognition  (Schraw  and  Dennison,  1994).  Philosophers  for  ages  have  been  intrigued  by  the  self-reflective  nature  of 
human  thought.  For  some,  it  is  the  behavior  that  is  the  basis  of  our  humanity.  Metacognitive  strategies  as  defined  by 
Flavell  (as  cited  in  Lin,  1994,  p.  489-490)  are  understanding  and  regulating  one's  own  cognitive  processes  in  order  to 
monitor,  direct  and  control  them.  Metacognitive  skills  include  perception  of  oneself  as  a learner,  an  awareness  of  the 
nature  of  a task's  components,  and  knowledge  of  when  and  how  to  use  effective  strategies.  These  factors  combine  to 
determine  which  tasks  learners  find  worthwhile  and  how  they  choose  to  engage  them  (Paris  & Winograd  as  cited  in 
Lin,  1994,  p.  490). 

Metacognition  becomes  increasingly  important  in  situations  of  heightened  learner  self-direction,  where 
learners  are  asked  to  decide  what,  how  and  when  to  explore  (Lin,  1994).  Examples  of  such  instructional  situations 
are  computer-based  hypermedia  simulations,  web-based  instruction,  and  asynchronous,  distance  learning 
environments.  These  environments  have  an  extended  complexity  and  a lack  of  structure  that  impose  increased 
responsibilities  and  cognitive  processing  requirements  on  users.  A lack  of  the  metacognitive  abilities  to  self-reflect, 
plan,  monitor,  evaluate,  and  adjust  one’s  own  cognitive  strategies  hinders  learning  under  these  conditions  (Horak, 
1991;  Blakey  and  Spence,  1990;  Ridley,  Schutz,  and  Glanz,  1992;  Lin,  1994). 

Today  the  application  of  metacognitive  research  has  become  useful  beyond  theoretical  models.  The  results 
are  metacognitive  strategies  that  students  can  be  taught.  Guided  practice  in  managing  their  own  experience  allows 
students  to  develop  the  metacognitive  strategies  needed  to  continue  to  direct  their  own  learning.  (Metcalfe  and 
Shimamura,  1994).  Strategies  include  connecting  new  information  to  former  knowledge,  deliberating  on  how  to 
select  what  to  learn,  planning  the  activity,  and  evaluating  what  is  learned.  Research  suggests  that  making  students 
aware  of  these  strategies  may  be  useful,  that  students  can  develop  strategies  on  their  own,  and  that  their  use  increases 
learning  and  performance  (Blakey,  1990).  Research  in  metacognition  has  followed  two  paths,  one  details  how  it 
develops  naturally  throughout  life  and  the  second  concentrates  on  the  training  potential  of  metacognitive  strategies 
(Lin,  1994).  Most  previous  research  on  metacognition  describes  the  phenomena  in  the  assimilation  of  complex 
information  rather  than  tasks  which  are  intended  to  produce  personal  or  novel  products.  This  study  was  situated  in  a 
studio  art  class  and  was  different  than  most  research  and  provided  a unique  context  for  metacognitive  strategies 
training. 

This  study  focused  on  training  students  in  the  use  of  metacognitive  strategies  and  the  effect  on  their  use  of 
metacognitive  processes  or  activities  in  self-directed  learning.  The  training  was  designed  into  the  instructor- led 
lessons,  which  focused  students'  attention  on  planning,  reflection  and  evaluation. 

The  long  range  goals  of  this  research  are  to  find  what  effect  training  metacognitive  strategies  will  have  on 
performance  and  future  use  of  metacognitive  processes,  as  well  as  to  identify  a method  of  training  metacognitive 
strategies  that  is  highly  effective.  Indicators  of  effect  will  be  the  performance  on  the  comparison  between  students' 
first  attempt  and  final  attempt  at  an  open  assignment. 

The  primary  expectation  of  this  study  was  that  students  who  have  or  develop,  and  then  use  metacognitive 
strategies  would  improve  performance  in  making  photographs.  Regardless  of  the  source  of  the  student's 
metacognitive  skills,  whether  gained  prior  to  or  during  the  course,  it  was  expected  that  using  strategies  to  monitor 

171 


what  they  know  would  increase  the  depth  of  their  knowledge  and  determination  to  make  images  which  were  better 
than  earlier  attempts.  We  hope  that  such  determination  supported  by  metacognitive  ability  would  continue  beyond 
the  course  as  students  develop  into  life  long  learners. 

Demands  that  are  put  on  students  to  be  self-reflective  and  creative  also  require  them  to  be  aware  of  their 
own  conceptual  underpinnings.  It  is  the  context  of  complex  learning  situations  that  puts  pressure  on  students  to 
develop  an  awareness  of  metacognition.  Since  the  course  content  was  complex  and  demanded  constant  improvement 
over  previous  work,  it  was  expected  that  post  treatment  surveys  would  show  significant  differences  to  prior  condition 
surveys  of  metacognitive  awareness. 

Method 

Participants 

During  the  fall  term  of  1998,  60  students  in  a sophomore  photography  class  for  non-art  majors  at  a state 
university  made  up  a sample  of  convenience  for  this  study.  There  were  22  males  and  38  females.  There  were  no 
graduate  students  and  all  but  4 students  were  under  25.  Participants  were  present  for  at  least  80%  of  all  classes.  The 
research  conformed  to  the  guidelines  set  forth  by  the  human  subjects  committee  of  the  university. 

Measures 

The  first  item,  a Metacognitive  Awareness  Inventory  (MAI)  (Appendix  A),  was  used  twice  as  a before  and 
after  comparison.  It  was  administered  the  first  time  in  the  second  week  and  a second  time  10  weeks  later.  The  MAI 
survey,  designed  and  tested  by  Schraw  and  Dennison  (1994),  provided  a reliable  test  of  metacognitive  awareness. 
The  research  conducted  by  Schraw  was  similar  to  our  testing  group  in  age  and  demographics.  The  survey  has  52 
statements  which  participants  reacted  to  by  marking  a Likert  scale  with  numbers  from  0 (never  true)  to  10  (always 
true).  The  statements  represented  two  component  categories  of  metacognition,  knowledge  and  regulation.  Within  the 
knowledge  component  were  statements  of  declarative  knowledge  (knowledge  about  self  and  about  strategies), 
procedural  knowledge  (knowledge  about  how  to  use  strategies),  and  conditional  knowledge  (knowledge  about  when 
and  why  to  use  strategies).  The  regulation  component  covered  planning  (goal  setting),  information  management 
(organizing),  monitoring  (assessment  of  one’s  learning  and  strategy),  debugging  (strategies  used  to  correct  errors) 
and  evaluation  (analysis  of  performance  and  strategy  effectiveness  after  a learning  episode). 

A self-report  survey  (Appendix  A)  was  administered  twice:  first,  after  students  completed  the  first 
assignment;  the  second  time  after  the  last  assignment.  The  survey  was  designed  to  capture  the  degree  and  type  of 
planning  and  self-reflection  students  did  before  and  during  the  preparation  of  the  work.  The  survey  reflects  the 
strategies  we  used  to  embed  training  in  context.  The  survey  questions  were  written  to  gather  data  and  to  be  a learning 
activity.  This  activity  shows  students  the  questions  they  should  be  asking  themselves  to  become  metacognitively 
aware  in  terms  of  planning  and  making  photographs. 

The  third  measure  was  a heuristic  used  to  compare  the  first  assignment  and  the  last  assignment  (Appendix 
A).  It  was  designed  to  measure  students’  performance  change  from  the  work  done  at  the  beginning  of  the  course  to 
work  done  at  the  end  of  the  course.  Each  participant's  work,  completed  in  four  categories  of  subject  matter,  was 
evaluated  based  on  technical  skill,  subject  matter,  and  composition.  Two  judges  were  asked  to  measure  the 
improvement  (0-3)  from  the  first  assignment  to  the  last  on  each  skill  category  for  each  subject  matter  set  of 
photographs.  Zero  meant  there  was  no  positive  difference  between  the  work.  A score  of  one  meant  slight  change; 
two,  noticeable  improvement;  and  three  meant  significant  positive  improvement.  Students  received  12  different 
scores,  totaling  a possible  36  points,  from  each  judge. 

Materials 

Instructional  activities  and  student  materials  were  developed  from  several  sources  of  research  descriptions. 
Blakley  and  Spence  (1990)  identify  specific  strategies  for  developing  metacognitive  behaviors  that  were  helpful  in 
creating  definitions  and  descriptions  of  metacognition  for  students  as  “Knowing  what  you  already  know....”  as  well 
as  developing  guided  practice  self-reflection  and  interview  questions.  Their  recommendation  of  social  interaction 
and  guided  discussion  about  thinking  in  context  of  course  content  was  included  in  the  treatment  activities  as  a 
necessary  condition  for  the  training  of  reflective  processes,  as  also  suggested  by  Von  Wright  (1992),  Erickson  and 
Simon  (1993),  and  Sitko  (1998). 

Procedures 

Students  met  for  10  three-hour  sessions  in  which  metacognitive  training  was  embedded  into  the  course 
content.  Each  meeting  presented  the  content,  which  had  been  part  of  the  syllabus  for  several  years.  The  lessons  for 
this  study  had  been  modified  to  require  students  to  recognize  others’  use  of  metacognitive  skills  while  working,  and 
practice  the  skills  themselves.  For  example: 
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• Several  videotapes  of  famous  artists  were  shown  in  class.  Participants  were  asked  to  actively  watch  the  film  while  answering 
questions  related  to  the  artist’s  methods  on  worksheets  during  and  after  the  showing.  They  also  were  expected  to  take  notes 
when  the  tape  was  stopped  to  illustrate  specific  instances  when  the  artist  was  employing  their  own  metacognitive  strategies  such 
as  planning,  monitoring  and  knowledge  assessment. 

• During  the  midterm  critique  class,  where  several  students'  photos  from  past  classes  were  shown,  questions  were  posed  to 
practice  planning,  visualization  and  self  reflection:  "Where's  the  camera?;  Where  has  the  photographer  placed  the  viewer?; 
When  you  are  making  a photograph,  how  do  you  manipulate  the  placement  of  the  camera  to  create  this  viewpoint?" 

The  first  and  last  assignments  were  given  10  weeks  apart;  they  had  identical  requirements.  Students  were 
instructed  to  make  photographs  of  four  subject  matter  categories:  portraits,  still  life,  landscapes,  and  self-portraits. 
They  were  required  to  submit  two  examples  in  each  category.  Emphasis  was  put  on  planning,  designing,  and  making 
unique  images  for  the  rest  of  the  students  to  enjoy,  understand,  and  as  defined  in  class,  “be  able  to  make  their  own.” 

Between  the  first  and  last  assignment,  three  other  assignments  were  given  over  a period  of  five  weeks. 

These  assignments  were  completed  in  three  parts.  They  gave  the  students  practice  in  self-reflection,  image 
visualization,  planning,  and  assessing  what  they  knew  or  needed  to  know.  The  first  part  was  a handout  instructing  the 
participants  to  write  or  sketch  some  ideas  for  this  new  photograph.  A week  later,  students,  in  a collaborative  planning 
exercise,  were  interviewed  by  each  other  about  their  ideas  for  the  assignment.  The  interview  was  facilitated  by  a 
worksheet  of  guiding  questions;  notes  were  taken  and  returned  to  the  person  interviewed.  For  the  third  meeting,  the 
photographs  that  were  done  for  this  assignment  were  brought  in  for  a critique.  Every  student  gave  a written  critique 
of  someone  else’s  work  and  received  a critique.  This  3-step  procedure  was  followed  for  all  three  practice 
assignments. 

After  students  had  completed  the  three  specific  subject  matter  practice  assignments,  they  were  given  the 
final  assignment.  They  had  to  repeat  the  first  assignment  to  demonstrate  what  they  had  learned  in  the  course.  It  was 
given  in  the  context  of  a final  exam  and  meant  to  challenge  students  to  think  and  plan.  The  importance  of  doing  the 
best  they  could  was  emphasized.  Their  evaluation  would  be  based  on  improvement  over  the  first  assignment. 

Research  Design 

The  study  was  designed  as  a related  sample  and  data  was  collected  for  mixed  method  analyses.  No  thought 
was  given  to  using  a control  group  due  to  the  ethical  problem  of  withholding  skills  training;  therefore  all  participants 
were  given  the  same  treatment.  The  independent  variables  are  the  pre-tested  metacognitive  awareness  MAI  scores 
and  a series  of  metacognitive  strategy  training  sessions.  The  dependent  variables  were  the  performance  in  making 
photographs,  measured  by  the  change  in  performance  score,  and  the  use  and  acquisition  of  metacognitive  skills, 
measured  by  the  post  MAI  score  and  self-report  surveys. 

Descriptive  statistics  illustrate  a difference  in  the  means  collected  from  the  MAI  survey.  Inferential 
statistical  procedures  (t  test  and  Wilcoxon  signed  ranks  test)  require  random  samples.  Our  design,  based  on  a sample 
of  convenience,  could  not  make  inferences  to  populations  beyond  this  classroom.  We  understand  that  the  best 
approach,  according  to  Ludbrook  and  Dudley  (1998),  would  be  a permutation  test  but  the  time  and  software  required 
were  not  available  to  the  researchers.  The  self-reported  data  on  the  use  of  metacognitive  skills  in  completing 
assignments  at  the  beginning  and  the  end  of  the  course  were  used  for  qualitative  analysis.  The  change-in- 
performance scores  were  used  to  create  two  performance  groups  of  ten  students  each,  High  and  Low,  as  a means  of 
descriptive  comparison. 

Results 

Changes  in  Total  Mean  Score . 

Figure  1 shows  the  difference  between  mean  scores  for  every  subject  before  and  after  the  treatment.  Table  1 
is  the  mean  and  standard  deviation  of  the  pre  and  post  MAI  scores.  We  cannot  defend  the  use  of  statistical  tests  to 
draw  inferences  to  a larger  population  but  we  do  feel  that  the  descriptive  statistics  support  a rational  verbal  argument 
that  this  group  was  trained  in  and  developed  additional  metacognitive  skills.  To  understand  the  effects  of 
metacognitive  strategies  training  on  changes  in  the  subjects’  performance  we  must  accept  the  reported  increase  in  the 
mean  of  metacognitive  use  is  due  to  the  embedded  treatment  in  the  course  content.  Qualitative  analysis  of  students’ 
changes  in  attitudes,  strategies,  and  self-reflective  reports  offer  additional  evidence  that  students’  thinking  and 
learning  strategies  for  completing  this  photography  were  affected  by  the  treatment  embedded  design  of  the  course 
content. 
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Figure  I.  Differences  in  Mean  MAI  Scores 


Comparison  of  Meanscores  of  pre  and  post  MAI  in  total 
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Table  /,  Mean  and  Standard  Deviation  of  Total  MAI  Scores 


N=60 

Mean 

SD 

Pre  MAI 

6.50 

1.12 

Post  MAI 

6.80 

1.18 

Self-report  surveys  Comparing  Assignments  1 & 8. 

Students'  self-report  surveys  were  analyzed  by  first  selecting  the  ten  extreme  examples  of  High  and  Low 
performing  students.  The  High  or  Low  category  reflects  the  level  of  change  of  performance  score.  The  High  score 
represents  a change  in  performance  score  of  33  or  more,  while  the  Low  group  had  scores  of  16  or  less,  from  a 
possible  score  of  72.  These  analyzes  (Appendix  B)  compare  in  two  ways  the  responses  of  these  students  to  the 
questions:  "What  skills  were  necessary  to  do  good  work?  . . . Did  you  have  them?"  and  "Describe  the  steps  you  took 
to  do  the  assignment."  The  first  compares  the  responses  on  assignment  1 and  8 for  each  student,  showing  individual 
change  from  the  beginning  to  the  end  of  the  course  and  how  this  change  differs  for  the  Low  and  High  performing 
students.  In  the  second  analysis,  responses  were  coded  into  metacognitive  categories  to  compare  the  use  of  these 
strategies  in  the  two  groups. 

The  comments  of  students  with  high  change  scores  tend  to  show  increased  confidence  in  their  work  and  a 
gained  awareness  of  the  skills  and  attitudes  required  to  make  good  photographs.  The  comments  of  the  lower  scoring 
students  tend  to  reflect  little  change  in  the  understanding  of  the  skills  needed  for  good  photography,  or  an  accurate 
awareness  of  their  actual  ability  to  make  photographs. 

For  example,  when  asked  "What  skills  were  necessary  to  do  good  work?",  nine  of  ten  high  performing 
participants  used  words  that  addressed  their  thinking  skills  rather  than  just  technical  information.  The  descriptors 
used  by  these  participants  for  necessary  skills  included  these  types  of  terms: 

• manipulating  meaning 

• determine  what  was  fitting,  interesting, 

• ability  to  look  at  photos  critically 

• use  different  perspectives 

• remove  things  to  simplify 

• have  patience,  knowledge,  creativity 
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High  performing  students  demonstrated  an  objective  look  at  their  recently  acquired  skills.  In  some  cases 
expressing  what  they  had  learned  and  what  they  still  needed  to  learn.  Participant  4 first  took  photographs  intuitively, 
’’doing  what  I felt”  but  later  described  necessary  skills  as  being  able  to  ’’determine  what  was  fitting,  interesting.” 
Participant  65  knew  at  first  that  she  did  not  know  or  have  the  ’’skills  to  do  good  work”  but  later  describes  feeling 
’’more  confident”  and  the  "skills  necessary  to  do  good  work  are  dedication,  acceptance  of  failure,  perseverance,  and 
creative  imagery.  Another  participant  (69)  used  the  same  words,  ’’patience,  knowledge,  creativity"  to  describe  skills. 

In  contrast,  the  descriptors  used  by  low  performing  participants  for  necessary  skills  included  such  terms  as: 

• attention  to  technical  details 

• know  your  f/stops  and  speeds 

• time,  patience  and  money 

• a good  eye  and  ability  to  work  the  camera 

These  participants’  first  comments  imply  that  they  consider  these  skills  as  things  that  have  not  yet  been 
given  to  them,  rather  than  something  to  pursue.  Six  low  performing  participants,  after  doing  assignment  8,  continue 
to  describe  necessary  skills  as  technical  ability,  like  "know  your  f/stops  and  speeds”  and  "focus,  aperture,  shutter 
speed”.  High  performance  students  used  different  terms:  "need  more  awareness  of  composition";  "tried  to  visualize 
pictures  I knew  were  possible";  "take  into  account  the  lighting,...  removing  things  from  the  picture  to  simplify  it.“  In 
both  groups,  the  comments  illustrated  a self-evaluation  but  the  key  difference  was  the  characterization  of  ’’skills"  for 
technical  (psychomotor)  goals  and  thinking  (cognition)  goals. 

When  responding  to  the  second  self-report  prompt  "Describe  the  steps  you  took  to  do  the  assignment."  the 
comments  of  both  groups  tended  to  be  more  similar.  One  High  group  student  describes  the  steps  for  the  first 
assignment  much  like  the  others  in  both  groups  ”...  looked  around  and  took  what  I thought  was  interesting." 
However,  that  student's  description  of  the  last  assignment  changed  to  ’’  Decided  to  photograph  things  there  that  have 
meaning  to  me."  Other  comments  more  evident  in  the  High  group’s  responses  for  the  second  assignment  relate  to 
photographs  being  "posed,  set  up,  staged"  and  many  comments  from  this  group  also  discuss  the  selection  process  as 
part  of  the  steps. 

After  participants'  comments  were  classified  according  to  metacognitive  components,  we  found  both  groups 
made  the  same  comments  related  to  the  first  assignment  in  the  planning  and  monitoring  categories.  In  planning,  both 
groups  characterized  their  behavior  as  "spent  a week  thinking,”  "planned  in  my  head,"  "thought  about  what  I liked." 
For  monitoring,  both  groups  made  similar  comments,  such  as  "photographed  what  I liked  that  applied,"  "took 
pictures  of  places,  people,  or  things  that  I had  been  admiring." 

The  low  performing  students  seemed  to  repeat  their  comments  from  the  first  assignment  when  describing 
assignment  8.  Most  of  their  comments  were  still  mentioning  "thought  about  subject  matter”  and  "thought  about  what 
would  be  better."  Two  participants  chose  to  suspend  any  planning  or  monitoring  and  let  chance  take  its  course.  They 
put  themselves  in  situations  and  relied  on  intuition  to  find  things  interesting  enough  to  photograph. 

The  difference  becomes  apparent  when  high  performing  students  report  how  it  is  "interesting  to  now  look  at 
similar  subjects  in  such  a different  light,"  Other  High  participants  said  "thought  about,  tried  to  visualize,  pictures," 
"had  ideas  for  pictures,"  and  "went  out,  tried  to  make  what  I visualized."  It  is  these  comments  that  illustrate  a change 
from  their  early  thoughts  about  what  makes  a good  picture  to  using  their  experience  as  interpretation. 

Discussion 

The  study  found  that  training  students  to  use  metacognitive  strategies  can  affect  learning  and  performance. 
The  results  also  show  that  students  who  improve  their  level  of  awareness  of  metacognition  also  apply  the  skills  to 
their  problem  solving  in  more  direct  content  specific  ways  than  those  who  do  not  change,  regardless  of  the  initial 
level  of  their  awareness.  The  implication  for  designers  of  instruction  is  that  instructional  strategies  for  practice  of 
content  should  be  embedded  with  exercises  that  also  improve  metacognitive  skills.  Students  should  be  given 
activities  which  require  them  to  become  aware  of  what  they  know,  plan  what  they  need  to  leam,  and  monitor  their 
strategy  choices,  so  they  can  be  self-directed  learners. 

One  of  the  most  significant  findings  of  this  research  is  the  value  of  the  embedded  design.  Embedded  design 
requires  that  metacognitive  strategies  be  related  directly  to  course  content.  For  the  instructional  designer  this  means 
developing  strategies  that  deliver  course  content  which  also  address  components  of  metacognitive  awareness. 
Activities  such  as  presenting  examples  of  experts  in  the  field  using  self-reflection  to  solve  problems,  writing  work 
plans  and  journals,  shared  dialog  in  determining  workable  solutions  to  given  problems  and  collaborative  critique  of 
work  are  some  examples  of  embedded  metacognitive  practice. 

As  learning  strategies  and  environments  become  more  self-directed  in  nature,  instructional  designers  must 
develop  ways  to  bring  these  theories  into  practice  to  help  learners  be  successful.  The  results  found  in  this  study 
conclude  that  embedding  metacognitive  practice  in  content  learning  activities  helped  students  become  more 
proficient  at  guiding  their  own  learning. 
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Another  important  observation  was  that  some  students  who  reported  high  metacognitive  awareness  did  not 
necessarily  report  their  use  in  doing  coursework.  Metacognitive  skills  need  to  be  practiced  like  any  other  skill  in 
order  to  become  effective.  Students  must  be  motivated  to  apply  metacognitive  strategies,  which  always  require  extra 
time  and  effort.  This  implies  that  designers  should  embed  not  only  metacognitive  practice  within  content  delivery  but 
motivating  strategies  as  well.  Future  replications  of  this  study  will  look  at  motivation  as  part  of  the  embedded  design. 
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Appendix  A 

The  measures  in  this  appendix  are  the  Metacognition  Awareness  Inventory,  the  student  self-report  survey  for  assignment  1,  and  the 
heuristic  used  to  measure  students’  change  in  performance. 

Name 

Please  read  the  statement  and  consider  how  much  you  agree  with  it... 
then  mark  the  scale  to  the  right  (0)  means  total  disagreement  10  means  total  agreement . 

0 10 

1 . I ask  myself  periodically  if  I am  meeting  my  goals.  (M) 

2.  I consider  several  alternatives  to  a problem  before  I answer. 

3.  I try  to  use  strategies  that  have  worked  in  the  past. 

4.  I pace  myself  while  learning  in  order  to  have  enough  time. 

5.  I understand  my  intellectual  strengths  and  weaknesses 

6.  I think  about  what  I really  need  to  leam  before  I begin  a task. 

7.  I know  how  well  I did  once  I finish  a test 

8.  I set  specific  goals  before  I begin  a task 

9.  I slow  down  when  I encounter  important  information 

10.  I know  what  kind  of  information  is  most  important  to  leam. 

11.  I ask  myself  if  I have  considered  all  options  when  solving  a problem. 

12.  Iam  good  at  organizing  information 

13.  I consciously  focus  my  attention  on  important  information. 

14.  I have  a specific  purpose  for  each  strategy  I use. 

15.  I leam  best  when  I know  something  about  the  topic. 

16.  I know  what  the  teacher  expects  me  to  leam. 

17.  Iam  good  at  remembering  information. 

18.  I use  different  learning  strategies  depending  on  the  situation. 

19.  I ask  myself  if  there  was  an  easier  way  to  do  things  after  I finish  a task. 

20.  I have  control  over  how  well  I leam. 

21.  I periodically  review  to  help  me  understand  important  relationships. 

22.  I ask  myself  questions  about  the  material  before  I begin. 

23.  I think  of  several  ways  to  solve  a problem  and  choose  the  best  one. 

24.  I summarize  what  I’ve  learned  after  I finish. 

25.  I ask  others  for  help  when  I don’t  understand  something. 
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I I I I I I I I I 


26.  I can  motivate  myself  to  leam  when  I need  to. 

27.  1 am  aware  of  what  strategies  I use  when  I study. 

28.  I find  myself  analyzing  the  usefulness  of  strategies  while  I study 

29.  I use  my  intellectual  strengths  to  compensate  for  my  weaknesses 

30.  I focus  on  the  meaning  and  significance  of  new  information. 

31.  I create  my  own  examples  to  make  information  more  meaningful. 

32.  I am  a good  judge  of  how  well  I understand  something. 

33.  I find  myself  using  helpful  learning  strategies  automatically. 

34.  I find  myself  pausing  regularly  to  check  my  comprehension. 

35.  I know  when  each  strategy  I use  will  be  most  effective. 

36.  I ask  myself  how  well  I accomplished  my  goals  once  I* m finished. 

37.  I draw  pictures  or  diagrams  to  help  me  understand  while  learning 

38.  I ask  myself  if  I have  considered  all  options  after  I solve  a problem. 

39.  I try  to  translate  new  information  into  my  own  words. 

40.  I change  strategies  when  I fail  to  understand 

41 . I use  the  organizational  structure  of  the  text  to  help  me  leam. 

42.  I read  instructions  carefully  before  I begin  a task. 

43.  I ask  myself  if  what  I’m  reading  is  related  to  what  I already  know. 

44.  I reevaluate  my  assumptions  when  I get  confused. 

45.  I organize  my  time  to  best  accomplish  my  goals. 

46.  I leam  more  when  I am  interested  in  the  topic. 

47.  I try  to  break  studying  down  into  smaller  steps. 

48.  I focus  on  overall  meaning  rather  than  specifics. 

49.  I ask  myself  questions  about  how  well  I am  doing  while  I am  learning  something  new. 

50.  I ask  myself  if  I leam  as  much  as  I could  have  once  I finish  a task 

51.  I stop  and  go  back  over  new  information  that  is  not  clear. 

52.  I stop  and  reread  when  I get  confused. 
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For  this  exercise  you  will  review  how  you  did  Assignment  1. 


Name 


In  order  to  improve  the  way  you  make  photographs  it  is  critical  that  you  take  time  to  review  and  reflect  on  how  you  make  photographs.  Learning  how  to  ask 
and  answer  the  following  questions  is  as  important  to  improving  your  work  as  making  photographs. 

Doing  the  Assignment 


Describe  the  assignment  in  your  own  words.... 


Explain  the  purpose  as  you  understood  it.... 


Describe  the  steps  you  took  to  do  the  assignment... 


Planning  the  Assignment 


Did  you  plan  how  to  make  these  photographs?  Yes  / No 

Did  you  write  your  ideas  down  Yes  / No...  or  sketch  the  composition...  Yes  / No 

If  you  did  write  or  draw  your  plan,  did  it  help  you  make  the  images?  Yes  / No 

As  for  the  technical  requirements  of  the  shooting.... 

Did  you  have  the  right  supplies?  ...  Yes  / No 

Did  you  take  the  pictures  all  at  once  or  over  several  sessions? 

How  many  rolls  of  Film  did  you  use?  

Did  you  design  a photograph  that  was  beyond  your  skills  or  expertise? 

Yes  / No 

Recall  how  Hockey  was  creating  and  then  designing  his  " joiner ”...  he  would  try  things  and  change  his  mind  and  talk  to  himself — when  you  were  taking 
your  pictures... 

Did  you  change  your  mind  while  taking  the  pictures?  ...  Yes  / No 

Did  you  talk  to  yourself  while  planning  or  taking  the  photographs?  ...Yes  / No 

Evaluating  the  Assignment 


Most  of  the  time  photographs  come  out  different  than  we  expect  them. 

When  you  first  saw  the  prints  from  the  processing  lab... 

Were  they  what  you  expected?  Yes  or  No 

Did  they  communicate  the  idea/meaning  you  had  intended?  Yes  or  No 
Did  they  give  you  ideas  for  new  photographs  that  you  want  to  make? 

Yes  or  No 


Did  you  redo  any  photographs  before  turning  them  in?  Yes  or  No 


How  did  your  evaluation  of  what  others  said  about  your  work  affect  how  you  think  about  doing  more  work?  In  other  words,  explain  how  you  will  use 
critiques... 


best  copy  available 


179 


ERIC 


171 


Appendix  B 


Student  ID: 

Total  Score: 

Name: 


Photo  Class  Assignment:  Measuring  Change  in  Performance 


No  Change  Some  Change  Definite  Change  Exceptional  Scores 

Change 


Portrait 


Technical 

0 1 

2 

3 

r~  _ 

Composition 

0 1 

2 

3 

Subject  Matter 

0 1 

2 

3 

Portrait 

Landscape 

Technical 

0 1 

2 

3 

V c . 

. Composition 

0 1 

, ' ' T . «/  ’ 

* 3 J ■ . 

*:-y 

Subject  Matter 

0 1 

2 

3 

Landscape 

Still  Life 

Technical 

0 1 

2 

3 

k . - 

Composition 

K 0 1 

■ r>  ■.  iK 

Subject  Matter 

0 t 

2 

3 

Still  Life 

Self  Portrait 

Technical 

0 1 

2 

3 

i.  • 

Composition 

0 : 1 

v 2__ 

- J 3 If  *\  • *, 

O li 

Subject  Matter 

0 1 

2 

3 

Self  Portrait 

Total  Score 


o 
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To  determine  how  metacognitive  strategy  training  differently  affected  metacognitive  processing,  the  data 
collected  from  40  of  the  self-reflection  reports  on  the  first  and  eighth  assignments  were  compared  in  two  ways.  The 
first  comparison,  Table  B1  shows  each  student’s  responses  to  two  specific  questions  asked  in  the  two  reports.  For 
Table  B2,  responses  were  coded  into  metacognitive  categories:  declarative  knowledge,  procedural  knowledge, 
conditional  knowledge,  planning,  managing  information,  monitoring,  debugging,  and  evaluating  strategies. 

Table  Bl.  Self  report  surveys:  Comparing  Assignments  1 & 8. 


What  skills  were  necessary  to  do  good  work?  . . . Did  you  have  them? 


High 

Assignment  1 

Assignment  8 

Low 

Assignment  1 

Assignment  8 

4 

Understood  requirements,  did 
what  I felt 

Determine  what  was  fitting, 
interesting. 

Shows  in  my  work 

1 

Definitely  Not. 

Skills  needed  were  good  selection 
and  composition  of  photos,  with 
attention  to  technical  details.  Yes 
I had  them. 

5 

Knew  what  I wanted  but  don't 
have  the  experience  to  manipulate 
the  camera  to  receive  the  desired 
results. 

Need  understanding  of  shutter 
speeds,  apertures,  and  light. ! 
could  have  used  more;  light  meter 
would  have  helped. 

8 

Yes,  after  taking  several 
pictures,  I remembered  how 
to  do  things  and 
experimented 

A good  camera,  a camera  you're 
used  to,  know  your  f/stops  and 
speeds. 

7 

I haven't  mastered  aperture  and 
speed,  so  used  automatic  settings. 
Decent  showing  of  ideas  & style. 

Knowledge  of  manipulating 
shutter  speed,  f-stop,  color  temp, 
meaning.  Yes. 

10 

Yes,  but  it  depends  what 
good  work  is.  It  was  fine  for 
my  second  project  but  far 
from  a gallery. 

It  was  very  important  that  you  had 
a good  idea,  of  how  to  work  your 
camera.  This  involves  items  such 
as  focus,  aperture,  shuttle  speed, 
etc.  Yes,  I feel  I have  a good 
understanding  of  the  skills  needed 
for  this  assignment. 

13 

Fair  job.  First  time  with 
adjustable  camera. 

Using  depth  of  field,  speed, 
lighting  contrast  knowing  what  to 
leave  in,  what  to  keep  out.  I feel  I 
was  better  at  using  these  skills  at 
the  end  of  the  semester. 

11 

Yes.  I seemed  to  manage 
taking  the  photos.  I think  I 
am  pretty  god  at  taking 
pictures. 

To  do  good  work,  one  must  be 
able  to  see  a situation  in  a 
photograph  and  make  it  look 
excellent  and  yes,  I did  have 
them. 

19 

I need  more  awareness  of 
composition.  I concentrated  every 
subject  square  in  the  middle  of 
the  field. 

Ability  to  look  at  photos  critically 
and  objectively  to  design 
photographs  that  were  better.  Yes. 

25 

No.  I don't  have  any  skill 
with  a manual  camera  and  I 
don't  quite  know  how  to  use 
it. 

Time,  patience,  money,  a need  to 
do  better.  I had  some  of  these. . . 
others  I lacked  in. 

51 

To  take  pictures  in  a way  that  the 
surroundings  in  the  picture  is 
interesting.  I think  I had  them. 

The  skills  in  the  class  that  1 
learned. . . the  different 
perspectives  of  things  that  I could 
photograph  enabled  me  to  do 
good  work. 

29 

Yes.  The  only  thing  I lacked 
was  time!  After  my  first  roll 
of  film  got  exposed  I had 
little  time  to  retake  photos. 

The  only  skills  needed  are  a good 
eye  for  a subject  and  ability  to 
work  the  camera.  Tm  not  great  at 
picking  subjects  but  I can  work 
the  camera  well. 

53 

I had  basic  knowledge  of  how  to 
use  my  camera  for  "snapshots". 
No  idea  what  type/genre  of 
photography  I preferred  and/or 
was  more  skilled  at 

I knew  more  than  I did  at  the  start 
but  still  need  practice 

45 

Creativity 

No  speed  skills. 

56 

I tried  to  visualize  pictures  I knew 
were  possible,  while  still  being 
something  that  I wanted. 

I needed  to  take  into  account  the 
lighting,  any  special  effects  I 
might  try,  removing  things  from 
the  picture  to  simplify  it,  etc.  I 
had  some  idea  of  these  skills 
before,  but  not  to  the  depth  I have 
now. 

46 

Yes,  I had  already  acquired 
the  basic  knowledge  needed 
to  do  the  assignment. 

In  order  to  do  this,  I needed  to  be 
able  to  quickly  determine  the 
camera  settings  and  also  be  able 
to  figure  out  what  made  good 
composition  - 1 got  better. 

65 

I did  not  have  all  the  necessary 
skills  to  do  good  work. 

Skills  necessary  to  good  work  are 
dedication,  acceptance  of  failure, 
perseverance,  and  creative 
imagery.  1 didn't  possess  all  this 
skills  when  I first  came  into  this 
class  but  now  I feel  more 
confident  that  I do  possess  them. 

64 

No,  I don't  think  I had  all 
the  skills  to  do  good  work.  I 
had  just  gotten  my  lsl 
manual  camera  & was  very 
unsure  of  it  & my  ability  to 
use  it.  I spent  an  entire  day 
taking  pictures  all  over  town 
& when  I went  to  pick  up 
the  developed  film  it  turned 
out  I had  loaded  the  film 
wrong  & didn't  get  a single 
picture. 

Creativity,  an  understanding  f 
your  cameras  functions,  an 
understanding  of  light,  space,  and 
your  subject.  Deciding  what  you 
are  trying  to  capture  1st. 

69 

In  general  my  photographs  were 
good  work,  visually  pleasing,  for 
beginning  photos. 

Patience,  knowledge,  creativity, 
work.  Yes. 

76 

For  the  most  pan,  some 
pictures  came  out  better 
than  others,  while  others 
came  out  not  as  I expected. 

To  plan,  see,  know  what  I wanted 
and  the  ability  to  create  images. 

- Sometimes. 
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Describe  the  steps  you  took  to  do  the  assignment 


Low 

High 

Assignment  1 

Assignment  8 

Assignment  1 

Assignment  8 

1 

Some  forethought  about 
who  and  what  I 
considered  photo  material 
- personal  choices. 

Waited  for  a trip  to  New 
Orleans  to  capture  one  of 
the  scenes 

Most  of  the  photos  were  taken  on 
a trip  to  Busch  Gardens.  I took 
many  rolls  with  this  assignment  in 
mind 

4 

I thought  of  a few  pictures  in 
my  head,  but  besides  those,  I 
just  kept  my  eyes  pen  and 
searched  for  subjects. 

See  things  not  obvious  to  regular  viewers. 
Sometimes  I just  got  lucky  with  what 
came  off  the  roll  of  film. 

8 

I looked  for  things  that 
intrigued  me. 

Took  a shot  of  it. 
Experimented  with  the 
same  shot. 

Picked  the  best. 

Studied  the  ones  messed 
up. 

Drove  out  to  Lake  Jackson, 
several  different  parks. 

Walked  the  trails,  looking  for 
something  that  caught  my  eye. 

5 

Figured  out  who  I wanted  the 
portraits  to  be  of. 

Just  looked  around  and 
thought  what  1 thought  was 
interesting. 

Ponder  what  I liked  so  I could 
then  know  what  I wanted  to 
shoot. 

Went  home.  Decided  to  photograph  things 
there  that  have  meaning  to  me. 

Walked  around  with  the  camera,  capturing 
some  things  by  accident. 

Others  I posed,  set  up. 

10 

Tried  to  picture  a scene 
that  would  be  easy  to 
produce. 

Picked  a person  and  a 
location 

Began  taking  pictures 

Planned  what  I wanted  to 
photograph. 

Made  a short  list. 

Went  about  photographing. 

7 

I went  out  to  Lake  Jackson 
for  things  to  photograph. 
Took  an  old  friend  out  and 
took  a couple  photos  of  her 
around  town,  home. 

Went  to  scenic  areas. 

Took  shots  I had  thought  about. 

The  objects  just  came  to  me  when  I was 
around  them. 

11 

Read  the  assignment 
Photographed  what  I 
liked  that  applied 

I kept  in  mind  what  the 
assignment  was. 

Tried  to  find  it. 

Took  the  picture,  developed,  and 
mounted  it. 

13 

Took  pictures  of  places, 
people,  or  things  that  I had 
been  admiring  for  some  time. 
The  self-portrait  was  more 
thought  out. 

Wandered,  exploring,  looking,  thinking 
about  compositions. 

Watched  for  good  lighting  opportunities. 
Staged  portrait. 

Cat  was  opportunity. 

25 

Took  my  camera 
everywhere  until  I found 
exactly  what  I wanted 

Looked  at  what  I had  done  before. 
Thought  about  what  would  be 
better. 

Attempted  to  do  it. 

19 

I thought  about  what  I would 
like. 

Made  a list  of  possible 
subjects. 

Took  far  more  than  necessary 
to  insure  acceptable  photos. 

Over  a month's  time,  I’d  have  an  idea  and 
set  it  up. 

Made  repeated  trips  to  Lake  Ella  for 
candid  shots. 

Took  more  than  necessary. 

Selected  the  best  examples. 

29 

Thought  about  things  that 
would  reflect  myself 
Took  the  photos 
Developed  them 
Chose  the  best 

I did  not  plan  the  subjects. 

I enjoy  just  taking  pictures  when  I 
see  an  opportunity.  Many  of  the 
photos  were  spontaneous. 

51 

Thought  of  what  kind  of  areas 
I wanted. 

Took  a trip  to  Tampa  and 
found  those  types. 

Some  things  I just  took 
pictures  as  I went. 

Selected  the  most  interesting. 

Thought  about  the  assignment. 
Through  the  weeks,  came  across 
situations. 

45 

Spent  a week  thinking. 
One  day  to  snap  a roll 

When  I was  free,  I just  grabbed 
my  camera  bag,  went  outside,  and 
took  pictures. 

Most  of  my  images  are  seen  every 
day  by  everyone. 

53 

Made  sure  1 understood  the 
assignment. 

Went  over  possibilities  in  my 
mind. 

Took  several  photos. 

Made  a photographing  trip 
around  town  and  campus. 
Chose  photographs  that  fit. 

Looked  back  at  1st  assignment  to  see  what 
to  do  differently. 

Looked  over  other  assignments  to  find 
preferences. 

Though  out  possible  ideas. 

Took  pictures  based  on  the  ideas. 

Other  things  seen  while  photographing. 
Picked  those  that  worked  best. 

46 

Planned  places,  people  or 
things 

Lighted,  distanced, 
focused  them 

Brought  my  camera  everywhere  I 
went  for  a month. 

Began  to  go  places  solely  for  the 
purpose  of  taking  the  pictures 

56 

Thought  about  what  would  be 
available  to  me:  lighting, 
subject,  etc. 

I went  out,  set  it  up,  shot  a 
couple,  trying  to  get  the 
angle,  shadows,  etc.  that  I 
wanted. 

Thought  about,  tried  to  visualize  pictures. 
Went  out,  tried  to  make  what  I visualized. 
Experimented  while  doing  this. 

64 

Think  about  my  subjects 
Drove  around  with 
camera. 

Got  them  developed 
Chose  which  pictures 

Looked  at  my  old  work  to  decide 
what  could  make  it  better. 
Decided  on  subjects,  began  to  get 
ideas 

65 

Planned  the  pictures  in  my 
head. 

Tried  to  take  those  and 
others. 

Took  more  random  pictures 
than  planned. 

Re-took  some. 

Reevaluated  my  own  work,  deciding  what 
I liked,  disliked. 

Interesting  to  now  look  at  similar  subjects 
in  such  a different  light.  Fully 
understanding  that  snapshots  are  not 
photographs. 

76 

76Planned  in  my  head. 
Pictured  what  & where 

Thought  about  subject  matter. 
Created  atmosphere  if 
controllable. 

Took  pictures. 

69 

Went  out  around  town. 
Took  various  pictures  from 
the  categories. 

Went  out  another  time  at 
night. 

Picked  out  the  ones  I liked. 

Had  ideas  for  pictures. 

Created  and  shot  them. 

Others  I just  saw  and  wanted  to 
photograph. 
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Table  B2.  Metacognitive  evidences  of  Assignments  1 and  8 for  both  groups 


■ ; : . / Low  Performance Group *.  # 

'5njfev».  ■ ’ />.  X::\:  • 

Component 

Description 

^Observed  Evidence 
(Assignment!) 

Observed  Evidence 
(Assignments) 

■Observed  Evidence 
(Assignment!) 

• Understood 

Declarative 

Knowledge 

• after  taking 

• Skills  needed 

Knowledge 

about  self  and 

several  pictures, 

were  good 

requirements. 

about 

I remembered 

selection  and 

did  what  I felt 

strategies. 

how  to  do 

composition  of 

"I  know..." 

things  and 

photos,  with 

• Knew  what  I 

experimented 

attention  to 

wanted  but 

technical 

don’t  have  the 

• I seemed  to 

details. 

experience  to 

manage  taking 

manipulate  the 

the  photos.  I 

• Time,  patience, 

camera  to 

think  I am 

money,  a need 

receive  the 

pretty  good  at 
taking  pictures. 

to  do  better 

desired  results. 

® I haven't 

• I had  already 

mastered 

acquired  the 

aperture  and 

basic 

speed,  so  used 

knowledge 

automatic 

needed  to  do 

settings.  Decent 

the  assignment. 

showing  of 
ideas  & style. 

• I need  more 

awareness  of 
composition. 

• To  take  pictures 

in  a way  that 
the 

surroundings  in 
the  picture  is 
interesting. 

•High  Performance  Group*  'V  p:4 ¥ 


Observed  Evidence  (Assignment  8) 


Need  understanding  of  shutterspeeds 
Apertures*  and  light. 

Knowledge  of  manipulating  shutter  speed, 
F-stop,  color  temp,  meaning. 

I knew  more  than  I did  at  the  start  but  still 
need  practice 

I needed  to  take  into  account  the  lighting, 
any  special  effects  I might  try,  removing 
things  from  the  picture  to  simplify  it,  etc. 

I had  some  idea  of  these  skills  before,  but 
not  to  the  depth  I have  now. 

Skills  necessary  to  good  work  are 
dedication,  acceptance  of  failure, 
perseverance,  and  creative  imagery. 

Patience,  knowledge,  creativity,  work. 
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F 


Procedural 

Knowledge 

Procedural; 

Knowledge 

•/*  . 

Knowledge 
about  how  to 
use  strategies. 

"First,  ...;  ■ 

then...”  . 

• . 1 don't  think  I 

had  all  the 
skills  to  do 
good  work.  I 
had  just  gotten 
my  1st  manual  . 
camera  & was  . 
very  unsure  of 
it  & my  ability 
. to  use  it.  I spent 
an  entire  day 
taking  pictures 
all  over  town  & 

;; : ; when  I went  to 
■ pick  up  the 

developed  film 
it  turned  out  I 
had  loaded  the 
• film  wrong  & 

didn't  get  a 
single  picture. 

• Created 
atmosphere  if 
controllable. 

It  was  very 
important  that 
you  had  a good 
idea,  of  how  to 
work  your 
camera.  This 
involves  items 
such  as  focus, 
aperture,  shuttle" 
speed,  etc. 

• To  do  good 
work,  one  must 
be  able  to  see  a 

; situation  in  a 
photograph  and 
make  it  look 
! ; excellent 

• The  only  skills 
needed  are  a 
good  eye  for  a 
subject  and 
ability  to  work 
the  camera.  I’m 
not  great  at  ; 
picking 
subjects  but  I 
can  work  the 
camera  well; 

• In  order  to  do 
this,  I needed  to 
be  able  to- 
quickly 
determine  the 
camera  settings 
and  also  be  able 
to  figure  out 
what  made 
good 

composition 

• Creativity,  an 
understanding 
of  your  cameras 
functions,  an 
understanding 
of  light,  space, 
and  your 
subject. 

Deciding  what 
you  are  trying 
to  capture  1st. 

• 1 had  basic 
knowledge  of 
how  to  use  my 
camera  for 
"snapshots". 

• No  idea  what 
type/genre  of 
photography  I 
preferred  and/or 
was  more 
skilled  at 

• 1 tried  to 
visualize 
pictures  I knew 
were  possible, 
while  still  being 
something  that  I 
wanted. 

• Using  depth  of  field;  speed,  lighting 
contrast  knowing  what  to  leave  in,  what  to 
keep  out.  I feel  I was  better  at  using  these 
skills  at  the  end  of  the  semester. 

• Ability  to  look  at  photos  critically  and 
objecti  vely  to  design  photographs  that  were . 

,,  better'. 

• The  skills  in  the  class  that  I learned...  the 
different  perspectives  of  things  that  I could 

. photograph  enabled  me  to  do  good  work. 

Conditional 

Knowledge 

Knowledge 
about  when  & 
why  to  use 
strategies 

"If...,  then  " 
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Planning 

, / 


■/••■■■■■-■  v . 

Goal  setting 
here's  how..." 

; : . ; ' 


: V SOIW  ;;.  ;-^,  .y 

MM&  forethought-#^* ' 
about  who  and 

/;//-' whjal/>  ' :/  ■ / 

' V considered  , 
photo  material  - 
//'•  personal  l ^ 

- choices. 

• Tried  to  picture 
;>/c  a-scene  that  / 

? v .•;  would  be  easy  x 
• / \ toVprbduce  // 

• ^ Thought  about  ■; 
/ //  \ things  ^ that  ^ 

would  reflect  . ; 

• v Spent  a week 

thinking.  ' 0 •’*'• 

• :, / Planned  places, 

: J j;-  people  or  things 

• ; Thinkabou  t roy 
subjects 

• Planned  in  my 

""  y-  h^;/;|/./:- /• 

Took  my 

'•■••/  camera  : ///:;/;/ 
eyeiywta^  7 
until  I found 
exactly  what  I 
.wanted 


:<  • To  plan;  see,  : 
///knowwhatl  -*/•  m 

,V.;;:K;;::>vanted and  they 
/'  //  ||ibiUty  to  create 

O0jmaiges.;  !.(■•'■ 

;•  Hann^whatl 
wanted  toy,;// : 
photograph; 

V- 

• '%  Thought  abbtit^ : ; 
what  would  be 

: Abetter; /•/'/',, •/> 

:>/:  Decided  on 
subjects,  began 
to  get  ideas 

' . Thought  about  / 
•/  subject  matter. 

•c  Began  to  go 
//x,:/;  places  solely;  for 
thepu^sepfi,/ 
//  - : taring  they///'; 
•/:/>';^<pictures; /;  //’  '■ 


• BvM*  I 

;//>  the  portraits  toy 

;z%5§zbe:pf.  ' //:;  •//// 

• •;/ 1 thought  about 

: 0 what  I would  0 

- . . > ...  -f  y.  «>> 


• Thought  of 
what  kind  o0 
7 • areas  I wanted. 

'J.  ’•  ‘ ' C -' . ' ‘yy*  1 . 5^*,  S. ^ 

: 0 Thought  about 


lighting, 
subject,  etc. 

• • Planned  th^/  . / 
: y / picture  in  my  < 
head 


■/  7y/7>f 
■V// 


Managing 

Information 


Organizing 
"Make  a list. 


Made  a list  of 

possible 

subjects. 


Monitoring 


Assessment  of 
one’s  learning 
and  strategy 

"How  am  I 
doing?" 


Read  the 
assignment  and 
photographed 
what  Hiked 
that  applied 

Lighted, 
distanced, 
focused  them 


Monitoring 


I kept  in  mind 

v I thought  of  a : 

what  the 

few  pictures  in 

assignment  was 

. my  head,  but 

and  tried  to  find 

//.  ..  besides  those,  I 

it  and  took  the 

just  kept  my 

picture. 

eyes  pen  and 

developed,  and 

searched  for 

mounted  it. 

subjects. 

Looked  at  what 

V Took  pictures 

I had  done 

of  places, 

before. 

people,  or 

things  that  I 

I did  not  plan 

had  been 

the  subjects  but 

admiring  for 

I enjoy  just 

some  time. 

taking  pictures 

when  I see  an 

• I went  out,  set  it 

opportunity. 

up,  shot  a 

Many  of  the 

couple,  trying 

photos  were 

to  get  the  angle. 

spontaneous. 

shadows,  etc. 

that  I wanted 

• Took  more 

random  pictures 

than  planned. 

185 

177 

Walked  around  with  the  camera,  capturing 
some  things  by  accident.  ■ //;/•• ; 0 : •/. 

Took  shots  I had  thought  about. 

Watched  for  good  lighting  opportunities. 

Took  more  than  necessary. 

Through  the  weeks,  came  across  situations 

Went  out,  tried  to  make  what  I visualized: 

Interesting  to  now  look  at  similar  subjects 
in  such  a different  light. 
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Debugging 

Strategies  used 
to  correct  errors 

"What  needs 
changing?" 

• Experimented 
with  the  same 
shot. 

• Studied  the 
ones  messed 
up. 

• I took  many 
rolls  with  this 
assignment  in 
mind 

• Attempted  to  do 
it. 

• Looked  at  my 
old  work  to 
decide  what 
could  make  it 
better. 

• Re-took  some. 

• Made  repeated  trips  to  Lake  Ella  for  candid 
shots. 

• Experimented  while  doing  this. 

• Reevaluated  my  own  work,  deciding 
what  1 liked,  disliked. 

Evaluating 

Analysis  of 

• Picked  the  best 

• : Sometimes  I just  got  lucky  with  what  came 

: performance 

off  the  roll  of  film. 

and  strategy 

• Chose  the  best 

effectiveness 

• Selected  the  best  examples. 

after  the 

;V v •’  . \ A . ;V  . 

learning 

• Fully  understanding  that  snapshots  are  not 

episode 

photographs! 

"How’ did  I do. 

J 

-V\". 

compared  to 

before  arid 

. 1 . - 

othere  • 

.y:y, 

o 
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Evaluating  Narrative  Simulation  as  Instructional  Design  for  Potential  to  Impact 

Bias  and  Discrimination 

Nancye  McCrary 
Joan  M.  Mazur 
University  of  Kentucky 


Abstract 

Simulation  exercise  as  instruction  has  a long  history  in  skill  development  involving  performance  tasks. 
Traditionally  physical  simulators  have  been  used  in  diverse  fields  such  as  military  flight  instruction  and 
industrial  training.  More  recently  paper  & pencil  simulation  exercises  have  employed  narrative  in 
instruction  aimed  at  solutions  to  work  place  problems , such  as  sexual  harassment.  This  evolution  has  led  to 
a refinement  of  principles  of  simulation  exercise  design,  as  well  as  evidence  for  specific  problem 
characteristics  that  indicate  potential  effectiveness  in  use  of  simulations.  Issues  such  as  racism,  anti- 
Semitism,  sexism,  disability , and  homophobia  present  instructional  challenges  due  to  the  covert  and  rigid 
nature  of  some  attitudes  and  beliefs.  This  paper  reports  on  the  development,  design,  and  initial  field  testing 
of  a narrative  simulation  focused  on  homophobia  and  adolescent  suicide.  Desired  learning  outcomes  that 
focus  on  attitudes,  beliefs,  and  biases  may  lend  themselves  to  further  evolution  of  narrative  simulation  in 
creating  contexts  in  which  to  examine  deeply  held  beliefs. 

Narrative  Simulation  as  an  Instructional  Design  Strategy 

Simulation  exercise  as  instruction  has  a long  history  in  skill  development  involving  performance  tasks. 
Traditionally  physical  simulators  have  been  used  in  military  flight  instruction,  industrial  training,  and  an  array  of 
other  performance-based  instructional  problems.  More  recently  paper  & pencil  simulation  exercises  have  employed 
narrative  in  instruction  aimed  at  accident  prevention,  farm  safety,  and  solutions  to  work  place  problems,  such  as 
sexual  harassment.  This  evolution  has  led  to  a refinement  of  principles  of  simulation  exercise  design,  as  well  as 
evidence  for  specific  problem  characteristics,  which  indicate  potential  effectiveness  in  use  of  simulations.  Issues  such 
as  racism,  sexism,  anti-Semitism,  disability,  and  homophobia  present  instructional  challenges  due  to  the  covert  and 
rigid  nature  of  some  attitudes  and  beliefs.  Group  discussion  is  one  powerful  way  of  beginning  to  examine  beliefs  and 
synthesize  new  information  regarding  biases.  Narrative  simulation  has  the  potential  to  initiate  such  discourse  by 
providing  less  threatening  contexts  in  which  to  think  about  and  express  deeply  held  beliefs.  As  a result  it  provides 
exposure  to  alternative  ways  of  thinking.  Desired  learning  outcomes  that  relate  to  attitudes,  beliefs,  and  biases  may 
well  lend  themselves  to  further  evolution  of  narrative  simulation  as  an  instructional  design  strategy. 

Based  on  the  work  of  Bruner  (1986),  Cole  (1994),  Howard  (1991),  Sarbin  (1986),  and  others,  this 
simulation  exercise  was  developed  as  instructional  material  for  presenting  content  on  homophobia  related  to 
adolescent  sexual  orientation.  "This  just  is!"  Jeff  s Story  uses  a narrative  approach  to  engender  discourse  and  to 
inform  regarding  this  difficult  and  prohibitive  topic.  It  specifically  deals  with  adolescent  sexual  orientation, 
homophobia,  and  potential  suicide.  It  is  a powerful  contemporary  culture  tale  that  offers  opportunity  for  situated 
presentation  of  information  (Lave  and  Wenger,  1991,  Schell  and  Black,  1997,  Young,  1993).  The  purpose  of  this 
paper  is  to  describe  the  design  and  preliminary  evaluation  of  this  innovative  narrative  simulation.  The  evaluation 
utilizes  data  obtained  through  expert  feedback,  exercise  field  tests,  individual  and  focus  group  interviews,  and 
observation  as  basis  for  informing  continued  development  of  this  design. 

“This  just  is”  Jeff  s Story  is  a narrative  simulation  designed  for  target  groups  of  parents,  helping 
professionals,  and  adolescents.  The  story  centers  on  a fourteen-year-old  boy  discovering  his  own  homosexuality  and 
struggling  with  problems  of  discrimination,  violence,  and  suicidal  thinking.  This  true  story  was  adapted  from  a web 
page,  developed  by  a parent  who  was  faced  with  these  issues.  The  title  "This  just  is"  was  adopted  from  a statement 
found  in  her  son’s  journal  after  his  suicide.2 

The  grand  scheme  was  to  explore  instructional  possibilities  relative  to  constructivist  theory.  Narrative 
simulation  provides  opportunity  to  teach  through  problem-solving  practice  in  real  world  environments  (Duffy  and 
Cunningham  in  Jonassen  Ed.,  1996).  The  overall  intent  was  concerned  with  exploring  instruction  designed  to  impact 
socially  constructed  attitudes,  creative  problem  solving,  and  independent  thinking  skills.  Simulation  exercise  with  its’ 
imperatives  for  “real  world”  learning,  veracity,  and  practice  before  the  fact,  as  opposed  to  retrospective  methods 
such  as  case  study  (Cole,  1997),  seemed  an  appropriate  place  to  begin.  The  narrative  simulation  approach 
increasingly  involves  the  user  in  the  story  by  virtue  of  question  & answer  decision  points  embedded  throughout. 


2 http://members. tripod. comZ-clavtolv/Bill’s  Story 


We  began  this  project  with  a broad  question  regarding  whether  instruction  can  be  effectively  designed  to 
decrease  bias  and  discrimination.  Questions  regarding  bias,  stereotyping,  and  attitudinal  change  appear  to  be  some  of 
the  most  challenging  issues  faced  by  contemporary  educators  (Boyer,  1995,  Vitz,  1990).  An  initial  single  focus  on 
homophobia  was  chosen  because  discrimination  towards  gays  is  so  pervasive  in  contemporary  American  culture 
(Helminiak,  1994).  Open  expression  of  homophobia  has  been  painfully  demonstrated  in  the  most  recent  well- 
publicized  case  of  Matthew  Shepard,  who  was  brutally  beaten  and  hung  on  a fence  to  die,  solely  because  of  his 
homosexuality  (Barrett,  1998).  Furthermore,  discrimination  towards  homosexuals,  in  most  cases,  is  still  legally 
condoned  in  this  country  (Plaster,  1998,  Stachelberg,  1998).  Relative  to  public  education,  Sandra  Prettyman  writes, 
"Creating  a forum  for  discussion  and  an  awareness  of  the  issue  [homophobia]  will  help  promote  student  safety  in 
schools  and  will  help  students  begin  to  recognize  and  critique  the  oppression  that  many  different  groups  of  people 
face  in  our  country  today  (1997,  p.  93)."  The  long-range  goal  for  this  project  is  to  develop  a series  of  narrative 
simulation  exercises  that  will  impact  various  kinds  of  discrimination,  both  in  pencil/paper  and  multimedia  formats 

Promoting  Tolerance  and  Inclusion  through  Instruction 

While  tolerance  and  inclusion  are  often  recognized  as  priority  issues  in  various  instructional  contexts 
(schools,  business  and  industry,  and  the  military),  problems  regarding  how  to  promote  sensitivity  to  difference  with 
regard  to  race,  culture,  religion,  gender,  disability,  and  sexual  orientation  persist.  Most  trainers  and  educators  realize 
that  addressing  these  issues  in  general,  rather  than  personal  terms,  has  proved  to  be  a superficial  and  ineffective 
approach  to  problems  that  have  serious  legal  and  moral  consequences.  It  has  been  particularly  difficult  to  address 
these  issues  in  engaging  and  meaningful  instructional  ways.  This  has  been  partly  due  to  the  fact  that  instructional 
materials  and  approaches,  designed  to  engender  personal  reflection  and  discourse  on  these  difficult  topics,  are 
limited,  often  nonexistent  (Prettyman,  1997). 

However,  there  is  an  emerging  body  of  qualitative  literature  that  seeks  to  address  these  issues.  The  1998 
journal  of  Qualitative  Studies  in  Education  11(2)  provides  an  editor’s  introduction  that  signals  "strong  support  of 
lesbian  and  gay  scholars  and  scholarship."  The  editors  say  "...we  feel  that  all  educators  need  to  make  known  publicly 
their  support  of  lesbians  and  gays  in  order  to  aid  this  group  in  their  struggles  (which  really  should  be  our  struggle, 
too)  to  gain  basic  human  rights."  William  Tierney,  author  of  the  book,  Academic  outlaws:  Queer  theory  and  cultural 
studies  in  the  academy,  says,  in  the  process  of  studying  a faculty  member  who  died  of  AIDS,  he  began  to  understand 
"...the  intense  dilemmas  that  individuals  face  in  coming  to  accept  those  parts  of  themselves  that  society  most 
vociferously  aims  to  deny..  .1  struggle  to  develop  pedagogies  that  enable  students  to  develop  their  own  voice  rather 
than  speak  with  one  voice  (Tierney,  1998,  p.  133)."  Robert  Rhoads  recently  completed  a "two-year  ethnographic 
study  designed  to  better  understand  the  coming-out  experiences  of  gay  and  bisexual  college  men  (Rhoads,  1997, 
p.8)."  Chasnoff  and  Cohen  have  published  an  instructional  film,  It's  elementary:  Talking  about  gay  issues  in  school 
(NewDay  Films,  1997),  which  is  reviewed  by  Prettyman  in  Educational  Studies  (1997,  pp.  92-98).  Catherine 
Raissiguier  (1997)  offers  a study  of  13  self-identified  homosexual  university  students.  Raissiguier  (1997)  notes  that 
while  earlier  studies  explored  the  "anomaly"  of  homosexual  attraction  and  behavior  it  is  relatively  recent  that 
homosexual  youth  have  been  studied  as  members  of  a collective  group  (pp.34-35).  These  studies,  related  articles, 
reviews,  and  editor's  notes  provide  a base  for  the  development  of  instructional  materials  and  media  designed  to 
promote  a deeper,  and  hopefully,  richer  understanding  of  all  types  of  difference,  particularly  that  of  sexual 
orientation. 

Affecting  Beliefs  and  Behavior  through  Narrative 

Walter  Fisher  (1995)  offers  well-articulated  foundation  for  use  of  narrative  to  integrate  the  didactic  and 
non-didactic  (affective)  aspects  of  instruction.  “Narration  (as  paradigm)  respects  reason.. .as  including  metaphor  as 
well  as  argument,  as  inextricably  bound  to  values,  and  as  historical  and  contextual  (p.173)."  Fisher  reminds  us  that 
objectivist  thinking  is  rooted  in  concepts,  methods,  and  hypotheses,  which  are  historically  grounded  in  myths.  He 
adds,  “objectivist  thinking  and  discourse  cannot  advance  without  recourse  to  imagination  (p.174)".  The 
implications,  for  purposes  of  this  project  are  to  design  instruction,  which  fills  the  space  beyond— the  gapping  hole-as 
it  were,  left  by  the  singular  focus  of  classical  science.  That  is,  instruction  designed  with  a primary  focus  on  values  in 
the  context  of  contemporary  culture.  Instructional  narrative  that  combines  didactic  information  and  narrative  may  be 
especially  powerful  with  regard  to  the  affective  domain  (Vitz,  1990). 

Henry  Cole  (1997)  has  organized  and  articulated  issues  regarding  instructional  narrative  simulation  through 
his  efforts  in  farming  and  mining  safety.  While  emphasizing  seemingly  traditional  performance  outcomes,  Cole  has 
also  provided  important  clarity  on  the  issue  of  impacting  learner  attitudes  and  beliefs.  Cole  constructs  solid 
theoretical  foundation  for  use  of  narrative  simulation  in  education.  He  states:  ‘The  robustness  of  the  life  stories 
people  live,  observe,  and  communicate  about  cannot  be  captured  in  a meaningful  way  with  only  the  abstract 
principles  of  paradigmatic  thinking  (p.332)."  One  important  contribution  Cole  makes  is  a narrative  conceptual 
model,  which  he  describes  as  “a  cultural,  cognitive,  and  behavioral  model... of  beliefs  and... behavior  (p.333)".  In 
this  model  culture  (stories  heard,  stories  lived,  and  stories  told)  contributes  to  cognition  (attitudes,  beliefs,  meaning,  . 
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and  knowledge),  which  in-tum  contributes  to  conduct  (behaviors,  actions)  that  results  in  consequences  (effects, 
outcomes).  While  the  process  is  linear,  insights  gained  through  it  often  fold  back  into  an  interplay  of  the  whole. 

Cole's  model  illustrates  the  relationship  between  narrative,  cognition,  behavior,  and  consequence.  One  point  that  he 
stresses  is  that  narrative  simulation  can  be  an  effective  means  to  vicariously  experience  consequences  and  may  even 
have  power  to  change  behavior  before  the  fact.  In  Cole’s  original  model  consequences  necessarily  follow  from 
narrative,  cognition,  and  behavior.  This  model  is  pertinent  to  this  development  project  with  one  exception.  That  is, 
we  would  add  some  indication  that  only  when  the  social  is  made  personal  do  consequences  necessarily  result  from 
this  process  (see  figure  1).  This  is  an  effort  to  move  Cole’s  model  a step  further  for  specific  consideration  of  issues  of 
social  values  such  as  discriminatory  behavior.  This  is  to  say  that  unless  the  cultural  collective  consequences  are  made 
personal  through  experience  (vicarious  or  real);  they  may  have  no  impact,  behaviorally  speaking,  on  individual 
attitudes  and  beliefs  underlying  discrimination.  ’’Bruner  (1986,  1990a)  notes  that  a universal  problem  in  education  is 
the  matter  of  translating  socially  relevant  information  into  information  that  is  personally  relevant  for  the  individual 
(Cole,  1997,  p.334).”  Narrative  form  has  the  power  to  make  the  socially  relevant  more  personal  for  the  learner 
(Bruner,  1986,  Cole,  1997,  Vitz,  1990). 

Figure  L Cultural,  Cognitive,  and  Behavioral  Model  of  Bias  and  Discrimination 


’’The  kind  of  meaning  the  narrative  conveys  about  human  existence  requires  the  use  of  discourse,  which  can 
be  differentiated  from  mere  collection  of  words  or  sentences.  A discourse  is  an  integration  of  sentences  that  produces 
a global  meaning  that  is  more  than  that  contained  in  the  sentences...(Polkinghome,  1988,  p.  31)."  In  addition  to 
personalizing  dilemmas  and  decisions,  narrative  also  has  important  potential  to  stimulate  group  discussion.  Bruner 
(1996)  writes  . .we  form  a representation  of  the  world  as  much  from  what  we  learn  about  it  through  others  as  from 
responding  to  events  in  the  world  directly  (p.165)."  This  concept  provides  an  important  frame  for  instruction  aimed 
at  impacting  the  affective  domain.  We  often  develop  attitudes,  beliefs,  and  biases  based  on  what  we  learn  from  others 
(individually  and  collectively)  rather  than  actual  experienced  events.  This  is  especially  true  with  regard  to 
homophobia  and  racism.  Usually  personal  experience  with  those  discriminated  against  is  limited.  Moreover,  even 
when  we  have  opportunities  for  such  experiences,  these  experiences  are  filtered  through  pre-existing  cultural  and 
personal  belief  systems.  Isolated  experiences  serve  to  either  confirm  pre-existing  notions  or  stand  as  rare  exceptions. 
Rarely  do  these  exceptions  impact  deeply  held  beliefs.  However,  since  experiences  that  result  in  bias  are  developed 
and  conveyed  largely  through  the  vehicle  of  narrative,  it  is  quite  appropriate  to  look  to  forms  of  instructional 
narrative  to  change  such  beliefs.  Sarbin  (1986)  contends  that  as  human  beings  we  make  moral  decisions  according  to 
narrative  structures.  As  such,  the  use  of  stories  has  an  enduring,  if  not  prevailing,  history  as  instructional  practice. 

Combining  Narrative  Structure  and  Simulation 

Some  important  findings  have  emerged  from  research  with  narrative  simulation  exercises.  Stories,  when 
used  in  this  manner,  provide  opportunity  for  critical  thinking  and  decision  making.  As  narrative  simulation  stories 
unfold,  dilemmas  develop  which  engage  the  learner  in  real  world  problems  and  relative  solutions.  The  narrative 
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exercise  requires  gathering  information,  interpreting  data,  making  choices,  remembering  and  utilizing  relevant 
information,  and  actually  interacting  with  the  plot  of  the  story  (Cole  1997,  Howard  1991,  Omer  1996,  Schell  and 
Black  1997,  Vitz  1990,  Young  1993).  The  learner  literally  becomes  a character  in  an  unfolding  tale.  As  such,  he  or 
she  is  more  likely  to  discuss  beliefs,  attitudes,  and  biases,  otherwise  considered  taboo.  Most  would  agree  that  bias 
and  discrimination  are  serious  problems,  which  tend  to  arrest  individual  and  cultural  growth  and  development. 
However,  the  tendency  of  human  nature  is  to  view  these  issues  as  other  people’s  problems  (Bennett,  1998).  As  an 
instructional  strategy,  situated  cognition  has  been  used  to  relate  subject  matter  to  the  concerns  of  learners  (Shor, 
1987).  Situated  learning  involves  immersing  learners  in  activities  and  environments  that  facilitate  ability  to  make 
meaning  (Lave  and  Wenger,  1991).  Situated  learning  typically  involves  four  guiding  principles:  (1)  learning  is 
grounded  in  everyday  activities;  (2)  knowledge  is  acquired  situationally  and  transfers  to  similar  situations;  (3) 
learning  is  the  result  of  a social  process  involving  certain  ways  of  thinking,  perceiving,  problem  solving,  and 
interacting;  and  (4)  learning  exists  in  complex  social  environments  made  up  of  actors,  actions,  and  situations 
(Anderson,  Reder,  and  Simon  1996).  Dilemmas  rather  than  content  drive  this  type  of  learning.  Certain  problems  are 
presented  that  serve  to  challenge  the  simulation  user. 

"This  just  is"  Jeffs  Story  combines  the  aesthetic  and  affective  power  of  narrative  with  the  situated  cognition 
of  simulation  by  re-creating  a real-life  situation  and  utilizing  strategically  placed  decision  points.  It  involves  the  user 
as  a character  in  an  unfolding  story,  providing  opportunity  to  think  through  problems,  respond  to  questions,  and 
resolve  dilemmas  as  the  story  progresses.  Additional  information  in  the  form  of  statistics,  news  media  excerpts, 
history,  and  commentary  is  provided  throughout  the  narrative.  These  appear  at  the  bottom  of  each  page  and  are 
distinguished  from  the  narrative  by  use  of  text  boxes.  Users  may  choose  to  read  any  or  all  of  this  additional 
information. 

Development  of  the  Narrative  Simulation:  "This  just  is!"  Jeff’s  Story 
The  Search  for  a Story 

The  imperatives  of  veracity  and  fidelity  for  simulation  effectiveness  led  to  an  initial  search  for  a true  story. 
While  it  is  possible  to  create  a narrative  with  truth-likeness,  it  seemed  that  it  would  be  more  powerful  to  use  an  actual 
contemporary  story.  The  power  of  contemporary,  true  story  narrative  to  support  reflection  and  change  has  been 
recently  documented  (Bliss  and  Mazur,  1996).  Additionally,  true  story  cases  combined  with  facilitated  discussion  of 
difficult  topics  can  create  conditions  for  conversation  and  community  (Mazur  and  Bliss,  1995).  While  Bruner 
(1990)  and  Grossman  (1992)  suggest  that  verisimilitude  or  plausibility  is  sufficient  in  narrative,  Phillips  (1994) 
argues  for  the  use  of  true  stories  ‘\vhen  something  important  hangs  in  the  balance  (p.  17).”  Moreover,  evidence 
exists  that  the  true  story  case  provides  for  intense  engagement  and  personal  involvement  in  the  story  (Bliss  and 
Mazur,  1995). 

Bill's  Story  appeared  in  the  form  of  a web  page  that  was  dedicated  to  prevention  of  teenage  suicide  and 
violence  towards  gays.  Bill  Clayton’s  mother  developed  this  web  page,  after  his  death.  She  was  contacted  and 
generously  agreed  to  assist  with  the  content  and  design  of  this  instructional  package.  Thus,  Gabi  Clayton  is  an 
important  collaborator  on  this  project,  offering  editorial  revisions  and  support  along  the  way.  For  example,  she 
suggested  that  the  use  of  the  term  lifestyle  in  the  original  answer  choices  is  a ’’trigger  phrase- lifestyle  has  been  used 
over  and  over  by  people  who  are  homophobic.  So  has  the  issue  of  choice”  (Gabi  Clayton,  1998,  in  e-mail  reply). 
Most  importantly,  she  provided  crucial  input  to  assure  maximum  fidelity.  Her  story  seemed  appropriate  for  use  is  this 
project  for  several  reasons.  It  is  a well-written  account  of  the  struggles  of  one  family  to  deal  with  adolescent 
homosexuality,  abuse  of  and  violence  towards  homosexuals,  depression,  and  teen  suicide.  More  importantly,  Bill’s 
Story  includes  vivid  accounts  of  dilemmas  and  decisions  faced  by  Bill  Clayton  and  his  family.  It  is  a true  story  in 
which  a family  faced  problems  head-on,  made  appropriate  decisions,  and  provided  more  than  adequate  support. 
However,  Bill's  Story  is  also  one  that,  despite  all  efforts  to  intervene,  ends  in  tragedy.  This  contradiction  is  the 
dynamic  diagonal  in  this  composition,  which  is  intended  to  inspire  robust  discourse. 

Adapting  the  Story  to  Narrative  Simulation 

The  next  issue  was  development  of  the  true  story  into  a narrative  simulation  exercise.  This  required 
transforming  it  from  a retrospective  account  to  an  unfolding  and  interactive  tale.  We  adapted  Cole’s  (1997) 
behavioral  model,  which  illustrates  the  relationship  between  narrative,  cognition,  conduct,  and  consequence.  He 
stresses  that  narrative  simulation  can  be  an  effective  means  to  vicariously  experience  consequences  and  may  even 
have  power  to  change  behavior  before  the  fact.  This  adaptation  goes  a step  further  (indicated  by  broken  lines)  to 
suggest  that  unless  the  cultural  collective  consequences  are  made  personal,  they  may  have  no  impact,  behaviorally 
speaking,  on  individual  attitudes  and  beliefs  underlying  discrimination.  [Insert  figure  1 about  here.] 

Following  the  work  of  Halpem  (1984)  and  Cole  (1997),  the  first  task  was  to  create  a qualitatively  different 
instructional  environment  that  would  encourage  critical  forethought  (simulation)  instead  of  informed  hindsight  (case 
study).  The  story  was  condensed  and  critical  decision  points  were  determined.  Each  decision  point  or  dilemma 
required  developing  corresponding  questions  and  answer  choices.  The  initial  questions  were  developed  and  reviewed 
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by  various  experts  in  the  fields  of  Educational  Psychology,  Curriculum  and  Instruction,  Instructional  Design,  and 
Ethics,  as  well  as  both  parents  and  gays.  These  reviewers  assisted  by  suggesting  important  editorial  revisions. 

Using  Qualitative  Field  Study  for  Content  Development 

During  this  initial  process  of  revision  it  became  increasingly  apparent  that  there  was  a need  for  the 
simulation  designer  (McCrary)  to  develop  a qualitatively  unique  and  deep  understanding  of  the  broader  issues  of  bias 
and  discrimination.  Various  techniques  of  qualitative  field  study  were  employed  to  broaden  the  designer’s  personal 
understanding  and  to  enrich  the  final  product.  The  final  tasks  in  development  of  the  instructional  components 
included  design  of  questions  and  instructional  response  choices,  as  well  as  supplementary  explanatory  information 
for  an  instructor’s  guide.  Due  to  the  fact  that  homophobia3  persists  as  an  “invisible  issue”  (Prettyman,  1997,  p.  92)  in 
American  contemporary  culture,  understanding  a range  of  possible  user  characteristics  and  designing  response 
choices  that  would  be  informative  and  meaningful  was  a daunting  task.  Qualitative  field  study  was  viewed  as  an 
appropriate  means  to  this  end.  A series  of  participant  observations  and  interviews  were  completed,  contributing  data 
specifically  on  homophobia,  as  well  as  the  larger  topic  of  discrimination,  including  racism,  sexism,  anti-Semitism, 
and  disability.  Two  major  reoccurring  themes  surfaced  throughout  these  formal  and  informal  observations  and 
interviews.  Those  were  religious  belief  and  unwillingness  to  claim  personal  bias.  From  a classical  instructional 
design  perspective,  religious  belief  and  denial  of  personal  bias  might  be  considered  as  important  entry-level 
characteristics  the  target  learner. 

For  purposes  of  this  design  effort,  religious  beliefs  may  be  no  less  important  than  preexisting  attitudes  and 
beliefs  about  worker  safety  or  gender  related  behavior  in  the  workplace.  Diana  Butler  Bass  (1998),  associate 
professor  of  religious  studies  at  Rhodes  College,  posits  that  in  the  cases  of  African  American  slavery,  the  subjugation 
of  women,  and  anti-Semitism.  ’’Christianity  developed  elaborate  theological  systems  based  on  a few  biblical  texts  to 
keep  people  subject  to  the  cultural  status  quo  and  male  authority  (D5)”.  It  follows  that  discrimination  towards 
homosexuals  may,  in  fact,  be  encouraged  by  such  theological  systems.  Additionally  cultural  experience  and 
background  may  contribute  to  miseducative  experiences  regarding  race,  gender,  and  minorities.  Eisner  references 
John  Dewey  in  describing  miseducative  experience.  “Miseducative  experience  arrests  growth  and/or  develops 
dispositions  towards  a domain  of  human  experience  that  limit  or  diminish  the  probability  of  growth  in  that  area. 
(Eisner,  1998,  p.99).” 

The  reoccurring  theme  of  the  refusal  of  those  observed  and  interviewed  to  claim  personal  prejudice  also 
raised  questions  and  design  considerations.  Most  informants  interviewed  were  quite  comfortable  discussing 
pervasive  discrimination  in  our  culture,  yet,  almost  without  exception,  viewed  it  as  the  behavior  of  others.  ’’But 
focusing  on  the  prejudices  of  others  leads  us  down  just  one  path  to  a hostile  and  divided  culture  as  each  group,  blind 
to  its  own  flaws  and  vulnerabilities,  comes  to  believe  that  it  holds  the  moral  high  ground  (Bennett,  1998).”  This 
reiterates  the  problem  in  education  of  social  versus  personal  relevance  (Cole,  1997,  see  figure  1).  Discriminatory 
behavior  usually  results  in  immediate  and  tangible  consequences  for  others,  rather  than  the  perpetrators  themselves. 
Therefore,  situating  the  learner  as  a character  in  an  unfolding  tale  through  the  use  of  engaging  questions  and  frank 
responses  may  be  the  most  crucial  function  of  narrative  simulation  for  this  type  of  instructional  design  effort.  "This 
just  is!"  Jeffs  Story  employs  this  function  in  situating  instructional  experience  in  an  environment  that  encourages 
reflection,  empathy,  and  compassion. 

Program  Description 

"This  just  is!”  Jeffs  Story  is  an  instructional  program  that  tells  a concise  story.  It  includes  an  embedded 
question  following  each  possible  decision  point  in  the  narrative.  After  each  question  several  possible  answer  choices 
are  listed,  as  well  as  space  for  write-in  answers.  Graphics  are  included  throughout  as  a means  to  appeal  to  aesthetic 
and  sensory  ways  of  understanding  and  remembering.  Each  page  of  narrative  includes  an  isolated  text  box  following 
the  story  segment  that  contains  pertinent  historical  accounts,  news  media  excerpts,  and  statistical  data.  A separate 
answer  discussion  key  is  attached,  which  includes  questions  and  answers  from  the  exercise  booklet.  Following  each 
possible  answer  choice  is  a subjective  commentary  relative  to  that  particular  answer.  The  answer  discussion  key,  as 
the  name  suggests,  is  designed  to  stimulate  group  discussion  and  individual  reflection.  Users  are  asked  to  compare 
their  answers  with  those  in  the  answer  discussion  key  and  read  the  accompanying  comments.  They  are  then 
encouraged  to  discuss  individual  opinions  with  the  group.  Since  this  instructional  package  is  intended  for  widely 
diverse  audiences  and  settings,  an  instructor’s  manual  is  provided.  The  instructor's  manual  includes  additional 
information  for  facilitating  participant  discussion.  It  provides  suggested  focus  for  discussion  and  clearly  outlined 
objectives.  This  manual  is  designed  to  prepare  and  guide  facilitators  through  the  instructional  process. 


2 Homophobia  has  been  defined  as  “irrational  fear  and  hatred  of  those  who  love  and  sexually  desire  those  of  the 
same  sex  (Pharr,  1988.  P.  1).”  Comparisons  have  been  drawn  between  homophobia,  racism,  and  anti-Semitism  in 
terms  of  discrimination  that  includes  verbal  and  physical  violence,  even  death  (Pharr,  1988) 
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Figure  2.  Sample  Narrative  Page 
" This  just  is!"  Jeffs  Story  Instructional  Simulation 


The  Beginning  of  the  Knd 

On  April  4,  1997,  Steven,  his  girlfriend,  Julie,  and  Jeff  were  walking  near  the  County  High  School  on  their  way  to  Julie’s  to 
watch  a video  they  had  rented.  Four  guys  from  their  high  school  were  cruising  by  the  school.  They  recognized  Jeff  and  followed 
the  three,  yelling  angry  obscenities.  “Queers,  faggots...” 

Steven,  Julie,  and  Jeff  tried  to  ignore  the  insults  and  threats,  deciding  to  cut  through  the  school  grounds  to  avoid  the  car 
following  them.  The  guys  in  the  car  parked  and  proceeded,  on  foot,  to  surround  Jeff  and  his  friends.  They  brutally  assaulted 
Steven  and  Jeff,  kicking  and  beating  them  unconscious.  It  was  a bright  spring  afternoon,  and  all  Julie  could  do  was  scream  at 
them  to  stop.  When  Steven  and  Jeff  regained  consciousness,  they  went  with  Julie  to  find  the  school  custodian.  He  called  the 
police  and  an  ambulance.  Both  boys  were  in  serious  condition  with  multiple  abrasions,  bruises,  and  broken  bones. 


Question  F:  As  Jeff ’ s parent,  you  are  now  faced  with  what  to  do  to  help  your  son.  How  will  you  proceed ? 

□ 41 . Get  Jeff  to  a psychiatric  hospital  immediately. 

□ 42.  Tell  Jeff  that  this  sexual  orientation  he  has  chosen  almost  got  him  killed  and  has  brought  nothing  but  trouble. 

□ 43.  Call  the  minister  at  your  church  to  get  his  advice. 

□ 44.  Tell,  Jeff  you  love  him  and  want  to  help  him  through  this  difficult  time. 

□ 45.  Talk  to  your  friends  and  try  to  get  their  advice. 

□ 46.  Try  to  find  Jeff  a nice  girlfriend. 

□ 47.  Make  an  appointment  with  a family  therapist  to  get  some  professional  help  for  you  and  your  son. 

□ 48.  Other 


Figure  3.  Sample  Didactic  Text  Box,  Graphic,  and  Original  Story  Quotation 
" This  just  is!”  Jeff  s Story  Instructional  Simulation 


A Bit  of  World  History 


“Who  belonged  in  a concentration  camp,  according  to  the  Gestapo?  Above  all  four  groups  of  people:  political  adversaries,  members  of 
inferior  races  and  inferiors  from  the  standpoint  of  race  biology , criminals,  and  asocials.  ...All  groups  of  prisoners  in  the  concentration 
camp  had  to  wear  external  markings  which  were  sewn  on  their  clothing,  namely  a number  and  a triangle  of  a certain  color  on  the  left  side 
of  the  chest  as  well  as  the  right  trouser  leg.  Red  was  the  color  of  the  political  prisoners..  .The  other  colors  and  designations  were  as 
follows:  Green  for  criminals. . . violet  for  Jehovah’s  witnesses,  black  for  asocials,  pink  for  homosexuals,  at  times  brown  for  gypsies.. . Jews 
wore  an  inverted  yellow  triangle  underneath  their  red,  green,  black,  or  other  markings,  forming  a star  with  six  points.”  (Kogon,  1946, 
cited  from  Duberman,  1989,  p.376) 

[Jeff  wrote  this  in  his  journal!  His  brother  found  it  when  he  was  going  through  Jeffs  belongings 

after  the  funeral.] 


Evaluation  of  "This  just  is!”  Jeffs  Story 
Focus  of  Field  Testing 

An  initial  field  test  of  the  narrative  simulation  was  conducted  with  two  user  groups.  These  included  a 
graduate  class  of  professionals  in  the  field  of  Higher  Education  Student  Development  at  a large  southeastern 
university  and  a small  group  of  ministers  certified  as  pastoral  counselors  working  in  a large  urban  area  in  the 
northeast.  Since  the  simulation  will  have  to  be  promoted  by  facilitators  or  those  engaged  in  raising  awareness  on  the 
issues  addressed  in  the  narrative  simulation,  these  groups  of  users  were  selected  for  initial  user  tests. 

The  design  features  prioritized  for  field-testing  were  usability,  impact  on  user  attitudes,  and  relevance  of 
didactic  information  embedded  throughout  the  exercise.  We  wanted,  first,  to  determine  the  overall  usability  for  ease 
in  following  instructions,  answering  questions,  and  cohesiveness  of  narrative  composition.  To  this  end  we  used  a 
participant  evaluation  that  was  designed  to  be  filled-out  immediately  after  completion  of  the  exercise.  Additionally 
participants  were  asked  to  fi  11-out  another  very  brief  evaluation  following  the  review  of  the  answer  discussion  key 
and  subsequent  group  discussion.  This  served  to  provide  specific  information  about  the  value  of  group  discussion. 
Cohesiveness  of  the  composition  was  determined  through  careful  analysis  of  participant  answer  choices,  with 
particular  attention  to  user  write-in  answers.  The  next  priority  was  to  collect  preliminary  data  on  the  potential  of  this 
exercise  to  impact  user’  attitudes  regarding  homosexuality.  This  was  done  through  a comparison  of  initial  participant 
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evaluations  with  a follow-up  evaluation,  which  was  completed  at  least  two  weeks  after  the  simulation  experience.  We 
considered  time  for  reflection  an  important  factor  in  attitude  change,  recognition  of  impact  on  beliefs,  and  ability  to 
articulate  such.  Finally,  we  wanted  to  determine  the  relevance  of  the  didactic  information  embedded  throughout  the 
narrative  simulation.  This  information,  appearing  in  text  boxes,  was  evaluated  through  overall  analysis  of  participant 
answers  and  comments  on  various  participant  evaluations,  as  well  as  comments  during  group  discussion. 

The  higher  education  development  professionals  used  the  simulation  exercise  during  one  graduate  class 
period,  while  the  ministers  used  it  during  a quarterly  ministerial  meeting.  The  narrative  simulation  packet  was 
distributed  and  users  were  instructed  to  work  through  the  simulation.  Immediately  following  individual  use  of  the 
simulation  packet,  users  filled  out  a brief  evaluation  that  included  questions  regarding  usability,  veracity,  informative 
aspects,  and  impact  on  attitudes  and  beliefs.  The  participants  then  worked  in  small  groups  to  discuss  their  responses 
and  the  narrative  itself.  Following  the  group  work,  users  were  asked  to  respond  to  several  questions  specifically 
regarding  the  impact  of  group  discussion.  Finally,  the  designer  debriefed  the  users  in  an  open-ended  group  discussion 
to  elicit  any  additional  feedback  not  addressed  by  the  evaluation  questionnaire.  We  also  attempted  to  contact  users 
after  several  weeks  in  an  effort  to  ascertain  residual  affects.  Although  the  initial  field  tests  include  fourteen  users,  we 
were  only  able  to  obtain  eight  returned  follow-up  evaluations.  The  follow-up  contact  occurred  at  least  two  weeks 
after  the  simulation  experience.  It  includes  ten  statements  similar  to  those  in  the  initial  evaluation. 

Formative  Evaluation  Results 

Table  1 summarizes  the  participant  evaluations  and  includes  percentage  results  for  eleven  questions 
regarding  the  individual  experience  and  four  questions  relative  to  the  group  discussion.  These  were  rated  from  one  to 
five  (5  = highest  agreement)  on  a Likert  scale.  As  part  of  our  analysis,  we  categorized  these  statements  under  six 
areas:  1)  veracity,  2)  general  usability,  3)  the  extent  to  which  the  user  felt  the  experience  was  informative,  4) 
cognitive  or  the  extent  to  which  the  exercise  provoked  thinking,  5)  impact  on  attitude,  and  6)  the  value  of  group 
discussion  in  the  exercise  process.  These  findings  are  summarized  and  reported  in  terms  of  general  agreement  with 
specific  evaluation  questions  as  indicated  by  user  choice  of  either  number  (4)  or  (5)  on  the  scale,  disagreement  by 
choice  of  number  (1)  or  (2),  and  those  choosing  number  (3)  were  viewed  as  neutral. 

Table  1.  Initial  User  Evaluation  Results  "This  just  is!”  Jeffs  Story  (n  - 14) 


Simulation  Evaluation  Questions 

Design 

Category 

Agree 

(#4,5) 

Disagree 

(#1,2) 

Neutral 

(#3) 

1 could  relate  personally  to  this  story. 

Veracity 

28% 

28% 

36% 

Jeffs  Story  was  realistic. 

Veracity 

86% 

0% 

7% 

The  questions  were  easy  to  answer. 

Usability 

57% 

43% 

0 

The  directions  in  the  exercise  are  easy  to  follow. 

Usability 

93% 

7% 

0 

1 found  the  additional  information  at  the  bottom  of  some  pages  appropriate  and  informative. 

Informative 

50% 

7% 

29% 

This  story  gave  me  new  information  about  homophobia  and  discrimination  towards  gays. 

Informative 

29% 

43% 

29% 

Prior  to  working  this  exercise  1 had  not  thought  much  about  discrimination  toward  gays. 

Cognitive 

0 

100% 

0 

What  1 learned  from  Jeffs  Story  will  help  me  respond  to  discrimination  towards  gay  youth. 

Cognitive 

50% 

14% 

36% 

The  questions  in  this  exercise  were  thought  provoking. 

Cognitive 

93% 

7% 

0 

1 found  Jeffs  Story  disturbing. 

Attitude 

79% 

0 

21% 

This  exercise  made  me  feel  uncomfortable. 

Attitude 

21% 

64% 

21% 

The  group  discussion  while  reviewing  the  answer  discussion  key  was  helpful. 

Collaboration 

86% 

0 

14% 

It  was  useful  to  hear  others  viewpoints  regarding  decision  points  in  the  exercise. 

Collaboration 

93% 

0 

7% 

The  group  discussion  caused  me  to  rethink  some  of  my  initial  responses. 

Collaboration 

71% 

14% 

14%  . 

The  group  discussion  enhanced  my  experience  with  this  exercise. 

Collaboration 

86% 

0 

14% 

Usability 

Two  specific  evaluation  items  related  to  general  usability.  Those  were  statements  regarding  following  the 
exercise  instructions  and  relative  ease  of  answering  the  embedded  questions  throughout  the  exercise.  93%  of  users 
indicated  that  the  instructions  were  easy  to  follow,  while  57%  agreed  that  the  questions  were  easy  to  answer.  During 
group  discussion  it  was  noted  that  one  reason  users  said  exercise  questions  were  difficult  was  because  they  struggled 
with  the  dilemmas  presented.  Written  comments  regarding  general  usability  were  as  follows.  ’’Good  exercise,  very 
interactive.”  ”It's  a wonderful  concept  and  the  tool  is  well  designed!"  "Excellent  investigative  tool." 

Veracity 

Two  statements  in  the  participant  evaluation  related  specifically  to  truth- likeness  of  the  narrative.  Although 
participants  were  told  that  this  exercise  was  based  on  a true  story,  we  wanted  to  know  whether  they  personally 
perceived  it  as  believable.  Thus  one  statement  concerned  the  extent  to  which  users  could  relate  personally  to  the 
story,  while  the  other  simply  stated  that  "This  Just  is!"  Jeffs  Story  was  realistic.  Only  28%  indicated  that  they  could 
relate  personally,  with  28%  claiming  inability  to  relate  and  36%  remaining  neutral.  However,  86%  of  participants 
indicated  that  the  simulation  was  realistic.  There  were  no  written  comments  specific  to  the  category  of  veracity. 
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Informative 

One  evaluative  statement  relative  to  the  informative  nature  of  the  simulation  dealt  with  appropriateness  of 
the  additional  information  provided  in  text  boxes  throughout  the  narrative.  A second  statement  in  this  category 
related  to  whether  the  story  itself  provided  new  information  about  homophobia  and  discrimination  toward  gays.  We 
wanted  to  know  if  such  a contemporary  story  was  familiar  to  users,  as  well  as  determine  whether  the  history,  news 
media  information,  and  statistics  provided  was  noticed  and  to  what  extent  it  appropriately  informed.  The  participants 
in  these  two  initial  field  tests  indicated  that  while  the  story  did  not  provide  new  information  on  homophobia  or 
discrimination  (only  29%  agreed),  the  text  box  information  was  appropriate  and  informative  for  50%  of  these  users. 
Two  written  comments  relative  to  this  category  were  especially  helpful  in  our  thinking  about  further  development. 
One  user  wrote,  "already  had  experience  with  homophobia".  Another  noted  that  the  historical  information  "seemed 
one-sided".  The  debriefing  discussion  clarified  this  issue.  Users  noted  that  more  geographically  relevant  examples 
would  be  pertinent. 

Cognitive 

The  statements  in  this  category  pertained  to  users'  sense  of  their  own  thinking  process  with  regard  to  the 
issues  at  hand.  These  were  designed  to  get  a sense  of  whether  participants  felt  they  had  thought  about  this  topic 
previously  compared  to  the  extent  to  which  this  experience  would  provoke  cognitive  processing.  We  also  used  a 
statement  about  whether  users'  felt  that  the  thinking  stimulated  through  the  exercise  would  impact  personal  behavior 
relative  to  discrimination.  The  issue  of  whether  participants  had  previously  thought  about  discrimination  towards 
gays  reflected  100%  disagreement.  That  is,  all  users  believed  they  were  aware  of  this  issue  before  participating  in  the 
field  test.  50%  indicated  that  this  experience  would  ultimately  change  their  behavior  with  regard  to  discrimination 
towards  gays,  and  93%  said  the  narrative  simulation  was  thought  provoking. 

Affective 

This  category  consisted  of  two  statements  intended  to  understand  the  extent  to  which  this  experience  was 
disturbing  and  made  participants  feel  uncomfortable.  Our  agenda  here  was  to  produce  some  way  to  begin  to 
determine  optimal  balance  between  necessary  discomfort  for  narrative  effectiveness  (cognitive  dissonance)  and 
residual  negative  feelings  as  a result  of  undue  discomfort.  Interestingly  enough,  79%  of  users  said,  ' 'This  Just  is!" 
Jeffs  Story  was  disturbing  and  only  21%  claimed  that  it  made  them  feel  uncomfortable. 

Group  Discussion 

We  felt  that  one  of  the  most  important  factors  for  productive  impact  of  this  instructional  design  would  be 
group  discussion.  In  that  spirit,  we  emphasized  collaboration  by  including  four  statements  regarding  group  process  in 
this  instructional  experience.  These  statements  had  to  do  with  whether  group  discussion  was  helpful,  whether  it  was 
useful  to  hear  others  viewpoints,  whether  group  discussion  caused  rethinking  of  initial  responses,  and  whether  group 
discussion  provided  overall  enhancement  of  the  experience.  Agreement  on  these  issues  was,  respectively,  86%,  93%, 
71%,  and  86%,  leaving  only  a small  minority  of  participants  neutral  or  in  disagreement. 

Follow-up  Evaluation 

Table  # 2 shows  the  results  of  the  participant  follow-up  evaluation.  The  general  themes  of  these  statements 
reflect  issues  similar  but  not  identical  to  the  initial  instrument.  These  themes  included,  broader  understanding, 
stimulation  of  further  discussion  of  homophobia  and  discrimination  towards  gays,  utilization  of  information  retained, 
actual  change  in  behavior,  continued  thinking  on  the  subject,  change  in  attitude  after  reflection,  lingering 
disturbance,  increased  awareness  and  sensitivity,  retention  of  information  gained  from  group  discussion,  and  overall 
satisfaction  with  having  had  the  experience.  One  striking  difference  in  the  follow-up  results  and  those  of  the  initial 
evaluation  was  that  more  respondents  indicated  a neutral  position  on  many  of  the  statements  in  the  follow-up  scale. 
63%  of  participants  indicated  a broader  understanding  of  homophobia  after  the  simulation  experience,  50%  said  they 
had  discussed  these  issues  further,  88%  suggested  that  the  experience  continued  to  be  thought  provoking,  63%  said 
they  still  found  the  exercise  disturbing,  and  76%  were  glad  they  had  participated.  In  disagreement  with  the 
statements  63%  indicated  that  they  had  not  yet  been  able  to  utilize  the  information  gained  from  the  exercise  and  51% 
said  they  did  not  feel  differently  than  they  did  immediately  after  the  experience. 
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Table  2.  Follow-up  User  Evaluation  Results  "This  just  is!"  Jeffs  Story  (n  = 8) 


Design  Category 

Agree 

(#4.5) 

Disagree 

(#1,2) 

Neutral 

(#3) 

I have  already  been  able  to  utilize  the  new  information  presented  in  this  exercise. 

Behavioral 

13% 

63% 

25% 

What  I learned  from  Jeffs  Story  has  helped  me  respond  better  to  discrimination  towards  gay 

Behavioral 

38% 

25% 

38% 

youth. 

1 have  discussed  my  thinking  on  this  topic  with  others  since  working  the  exercise. 

Cognitive 

50% 

38% 

13% 

I find  this  experience  has  broadened  my  understanding  of  homophobia. 

Cognitive 

63% 

25% 

13% 

The  questions  in  this  exercise  continue  to  be  thought  provoking  for  me. 

Cognitive 

88% 

13% 

0 

After  reflection  I felt  differently  about  this  experience  than  I did  immediately  after  working 

Attitude 

38% 

51% 

13% 

the  exercise. 

I am  more  aware  and  sensitive  to  broad  issue  of  discrimination  since  my  experience  with 

Attitude 

26% 

26% 

50% 

Jeffs  Story. 

I am  glad  I had  the  experience  using  this  exercise. 

Attitude 

76% 

13% 

13% 

I still  find  Jeffs  Story  disturbing. 

Attitude 

63% 

25% 

13% 

I remember  more  from  the  group  discussion  than  the  actual  exercise. 

Collaboration 

25% 

38% 

38% 

Reflections  on  Instructional  Narrative  Simulation 

What  are  the  implications  for  the  use  of  narrative  simulation  to  address  difficult  topics  such  as  bias  and 
discrimination?  To  what  extent  can  the  use  of  qualitative  development  strategies  and  ethnographic  field  study 
methodologies  inform  the  design  of  such  instructional  materials?  The  results  of  preliminary  field-tests  are 
encouraging  in  several  areas.  First,  the  transformation  of  a true  story  case  into  an  instructional  narrative  simulation 
was  possible  and  indeed  achieved  some  of  the  desired  design  intentions  (character/circumstance  identification  and 
user  engagement).  Secondly,  embedding  didactic  information  along  with  narrative  informed  and  served  to  further 
frame  the  story  as  contemporary.  It  appears  that  the  embedded  didactic  information  accomplished  these  objectives 
without  eroding  the  flow  of  the  story  or  the  involvement  for  the  majority  of  users  in  this  group.  Third,  the 
combination  of  the  narrative  simulation  with  group  discussion  appears  to  have  potential  to  greatly  increase  the 
impact  of  the  experience.  This  is  particularly  important  because  it  is  precisely  the  lack  of  public  discussion  and 
disclosure  of  personal  belief  that  appear  to  be  key  issues  in  addressing  the  so-called  “invisible”  aspects  of 
homophobia  and  other  forms  of  discrimination.  Fourth,  This  experience  of  the  simulation  appears  to  have  affects 
beyond  the  initial  use  of  the  exercise,  as  indicated  by  responses  to  the  follow-up  evaluation.  Finally,  the  results  of 
this  initial  user  test  suggest  several  improvements  in  the  design.  Those  include  the  use  of  more  geographically 
diverse  didactic  information  to  promote  user  identification  and  revisions  to  the  instructor’s  manual  regarding  reaction 
to  user  responses. 

These  results  engender  a commitment  to  look  carefully  at  responses  to  questions  on  additional  tests  with 
other  user  groups  (parents,  adolescents,  etc.),  in  order  to  make  informed  judgments  regarding  retaining  questions  that 
are  never  selected  or  may  otherwise  detract  from  the  power  of  the  narrative.  Lastly,  through  the  design  process  for 
the  narrative  simulation,  we  have  found  the  use  of  qualitative  methods  in  the  development  process  has  much  promise. 
Such  field  study  immerses  the  designer  in  the  context  of  instruction  and  also  provides  rich,  contemporary  insights 
that  can  be  folded  back  into  the  design. 
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Abstract 

Student  assessment  refers  to  the  evaluation  of  students * performance  using  appropriate  methods.  The 
impact  of  student  assessment  on  learning  is  emphasized  by  many  educators.  However,  not  many  articles 
discuss  the  methodology  of  assessment  in  a problem-based  learning  approach.  The  purpose  of  this  paper  is 
to  review  evaluation  theories  and  techniques  in  both  the  medical  and  educational  fields  and  to  propose  an 
evaluation  theory  to  explain  the  condition  variables,  the  method  variables , and  the  outcome  variables  of 
student  assessment  in  a problem-based  learning  approach.  A categorization  scheme  of  assessment  will  be 
presented,  which  includes  a wide  range  of  evaluation  techniques  and  tools  to  assess  students'  behaviors 
and  products  in  a problem-based  learning  approach. 

Introduction 

Evaluation  in  problem-based  learning  (PBL)  includes  student  assessment  and  program  evaluation.  Student 
assessment  refers  to  the  evaluation  of  students’  performance  using  appropriate  methods.  The  impact  of  student 
assessment  on  learning  is  emphasized  by  many  educators.  However,  not  many  scholars  discuss  how  professors 
evaluate  students.  Swanson,  Case,  and  van  der  Vleuten  (1991)  note  that  “despite  recognition  of  the  importance  of 
assessment  among  problem-based  learning  advocates,  there  is  little  agreement  on  methodologies  for  assessment.” 

The  purpose  of  this  study  is  to  review  evaluation  theories  and  techniques  in  both  the  medical  and  educational  fields 
and  to  propose  an  evaluation  theory  to  explain  the  condition  variables,  the  method  variables,  and  the  outcome 
variables  of  student  assessment  in  a problem-based  learning  approach.  A categorization  scheme  of  assessment  will  be 
presented,  which  includes  a wide  range  of  evaluation  techniques  and  tools.  In  this  paper,  PBL  will  be  briefly 
discussed  in  respect  to  its  definition,  process,  and  intended  learning  outcomes.  Secondly,  the  theoretical  basis  of  PBL 
evaluation  and  the  categorization  scheme  of  assessment  will  be  presented. 

Problem-based  learning 

Problem-based  learning  is  a commonly  used  instructional  approach  in  today’s  medical  schools  in  the  United 
States.  The  advantages  of  PBL  make  it  a superior  way  compared  to  the  teacher-centered  lecture  approach.  The 
traditional  approach  was  criticized  as  placing  too  much  emphasis  on  memorization  but  not  enough  emphasis  on 
developing  thinking  skills.  Students  are  passive  learners  and  fail  to  transfer  their  knowledge  to  different 
environments.  Whereas,  PBL  fosters  students’  multiple  skills  and  allows  them  to  acquire  knowledge  in  the  process  of 
solving  problems.  In  this  section,  the  learning  process  and  intended  outcomes  of  PBL  will  be  reviewed. 

The  PBL  definition  and  process 

Problem  based  learning  is  defined  as  “the  learning  that  results  from  the  process  of  working  toward  the 
understanding  or  resolution  of  a problem”  (West,  1992).  In  a PBL  approach  the  students  are  active  learners.  Several 
authors  (Gallagher,  1997;  Savery  and  Duffy,  1996;  West,  1992)  describe  students’  tasks  in  the  learning  process. 
According  to  their  description,  all  students  in  a small  group  work  on  the  same  problem  presented  by  the  teacher. 

They  need  to  internalize  and  reason  through  the  problem  to  identify  the  learning  issues  and  define  the  objectives.  As 
tasks  and  work  schedule  are  assigned  to  each  group  member,  they  start  to  identify  resources  to  gather  information 
related  to  the  learning  issues.  Each  member  brings  back  and  reports  his  findings  for  the  whole  group  to  analyze  the 
data  or  to  shape  their  learning  issues  in  order  to  identify  the  solutions.  Once  students  synthesize  and  summarize 
findings  into  solutions,  they  need  to  conduct  an  assessment  to  see  if  all  the  objectives  have  been  satisfied  and  to 
evaluate  their  performance.  Finally,  they  receive  feedback  from  the  facilitator.  To  summarize,  the  tasks  students  need 
to  perform  are: 


■ Defining  problem  and  identifying  the  issues 

■ Framing  learning  objectives 

■ Determining  work  schedule 

■ Gathering  and  analyzing  data 

■ Presenting  and  discussing  learning  findings  related  to  learning  issues 

■ Synthesizing  and  summarizing  findings  into  solutions 

■ Justifying  solution  and  evaluating  performance 
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Goals  of  PBL 


PBL  approach  is  intended  to  foster  student's  skills  in  facing  their  future  learning  and  profession.  Students' 
goals  of  learning  include  metacognition  skills,  self-directed  learning  skills,  critical  thinking  and  problem  solving 
skills,  learners'  knowledge  acquisition  and  use,  collaborative  learning,  as  well  as  their  higher  motivation  and  positive 
attitude.  In  this  section,  these  goals  will  be  shortly  discussed. 

Metacognition 

According  to  Barrow  (1988),  metacognition  is  thinking  about  thinking.  It's  the  "executive  function  in 
thinking:  pondering,  deliberating,  or  reflecting  on  the  problem  or  situation;  reviewing  what  is  known  and 
remembered  about  the  kind  of  problem  confronted;  creating  hypotheses,  making  decisions  about  what  observations, 
questions  or  probes  need  to  be  made;  questioning  the  meaning  of  new  information  obtained  from  inquiry;  reviewing 
what  has  been  learned,  what  it  all  may  mean  and  what  needs  to  be  done  next,  etc."  An  expert  problem  solver  can 
monitor  and  control  their  thinking  process  to  attain  their  best  effectiveness. 

Self-directed  learning 

Since  knowledge  changes  rapidly,  PBL  approach  fosters  students'  self-directed  learning  skills  so  that  they 
will  automatically  continue  their  own  learning  for  the  rest  of  their  lives  (Barrow,  1988).  PBL  allows  students  to 
determine  what  and  how  to  learn  on  their  own  in  order  to  ensure  their  lifelong  learning  skills  and  the  ability  to  be 
effective  and  independent  learners. 

Critical  thinking  skills  and  problem  solving  skills 

According  to  Scheiman  and  Dell  (1989),  the  problem  solving  process  has  been  called  the  clinical  reasoning 
process,  which  refers  to  "the  cognitive  process  that  is  necessary  to  evaluate  and  manage  a patient's  problem."  In  a 
PBL  approach,  students  need  to  identify  learning  issues,  develop  strategies,  locate  information  relevant  to  the  issue, 
and  evaluate  their  solution.  This  entire  learning  process  helps  to  develop  their  problem  solving  and  thinking  skills 
(Savery  and  Duffy,  1996). 

Knowledge  acquisition , retention,  and  use 

Wilderson  and  Feletti  (1989)  claim  that  two  types  of  knowledge  learners  acquire  when  they  solve  problems 
are  concept  and  procedural  knowledge.  In  other  words,  they  are  knowledge  of  concepts,  principles  and  facts,  as  well 
as  the  procedure  for  how  to  use  them.  Hmelo  and  Ferrari  (1997)  note  one  goal  of  PBL  as  the  construction  of  an 
extensive  and  flexible  knowledge  base.  They  summarize  several  studies  and  conclude  that  students  in  PBL  learn  at 
least  as  much  social-studies  and  life-science  content  as  students  in  a traditional  classroom. 

Collaborative  learning 

PBL  facilitates  the  development  of  effective  collaborative  learning  practices.  Students  working  in  small 
groups  compare  their  thoughts  to  those  of  others.  They  develop  individual  perspectives  and  recognize  those  of 
others.  PBL  not  only  provides  students  with  the  opportunity  to  communicate  in  a social  setting  but  also  enables  them 
to  cooperatively  work  with  others  to  understand  different  views. 

Higher  motivation  and  positive  attitude 

In  a PBL  program,  students  are  active  learners  and  have  the  ownership  for  their  learning.  Therefore,  they 
express  a more  positive  attitude  and  higher  motivation  in  learning  than  do  students  in  traditional  programs. 

Need  for  an  evaluation  theory 

As  I reviewed  the  evaluation  theories  and  techniques  in  several  articles,  I found  that  some  authors  use 
different  terminology  to  refer  to  the  same  concepts.  In  addition,  there  are  controversies  in  their  view  of  different 
evaluation  techniques  and  tools.  Some  of  them  present  the  assessment  methods  that  are  only  applied  in  medical 
schools,  while  others  describe  their  experience  of  using  PBL  in  other  fields.  I synthesized  those  evaluation 
techniques  and  identified  gaps  or  conflicts  between  them.  As  a result,  I find  it  is  necessary  to  generate  an  evaluation 
theory  for  a problem-based  learning  approach  that  synthesizes  these  theories  and  techniques  in  a more  holistic  way 
and  provides  comprehensive  evaluation  techniques,  assessment  circumstance,  as  well  as  outcomes  to  which 
educators  and  future  researchers  can  refer.  In  the  following  sections  I present  my  theory  of  student  assessment,  which 
is  underpinned  by  several  assessment  theories  and  studies,  as  well  as  by  my  personal  experience. 
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Theoretical  basis  and  the  philosophy  of  assessment 

PBL  was  first  implemented  in  medical  education  in  the  early  1970's  and  gradually  was  applied  to  other 
fields.  Norman  and  Schmidt  (1992)  note  that  PBL  is  a learning  method  based  on  the  rationalist  tradition  and  is 
strongly  influenced  by  cognitive  psychology.  According  to  Schmidt  (1993),  cognitivists  regard  prior  knowledge  as 
the  determinate  of  the  new  information  to  be  processed  by  individuals,  and  knowledge  is  structured  in  memory.  The 
problem  presented  to  students  in  PBL  helps  students  to  activate  their  prior  knowledge  and  to  restructure  appropriate 
semantic  networks  in  order  to  solve  the  problem.  Based  on  the  epistemology  of  constructivism,  the  problem  in  a PBL 
program  works  as  a stimulus  for  authentic  activity.  Students  are  expected  to  reflect  on  their  activities  and  evaluate 
their  strength  and  weakness  (Duffy  and  Cunningham,  1996).  Duffy  and  Cunningham  consider  assessment  to  be 
ongoing,  rather  than  just  being  an  end-of-the-semester  rating.  Therefore,  assessment  must  be  in  the  learning  context 
and  can  not  be  separated  with  the  process  of  learning.  They  note  that  "the  abilities  to  self-assess  and  to  provide 
constructive  feedback  to  team  members  are  explicit  learning  goals,  and  this  is  not  only  an  assessment  process  but 
also  a learning  process.”  Talking  to  the  teacher’s  role  in  assessment,  Brooks  and  Brooks  (1993)  suggest  evaluating 
students  through  teaching,  participating  in  interaction,  and  observing  and  watching  student  work  with  ideas  and 
materials.  The  teacher  can  ask  questions  that  challenge  students  to  assess  themselves. 

The  evaluation  methods  presented  in  this  paper  cover  those  from  both  cognitivism  and  constructivism.  No 
matter  what  epistemology  a PBL  is  based  upon,  it  is  important  to  provide  a supportive  learning  environment  so  that 
students  are  safe  to  make  mistakes  and  to  fail  (Bridge  and  Hallinger,  1995). 

Categorization  scheme  of  evaluation 

Different  taxonomies  have  been  proposed  by  scholars  to  classify  evaluation  methods  in  PBL.  Glasgow 
(1996)  categorizes  assessment  strategies  into  three  areas  of  content,  process,  and  outcome.  According  to  him,  content 
assessment  is  concerned  with  knowledge  students  acquired.  Process  assessment  focuses  on  students’  abilities  to  apply 
knowledge  in  solving  problems.  Outcome  assessment  deals  with  the  products  created  by  students  that  may  involve 
original  knowledge  or  a new  application  of  knowledge.  Similar  to  Glasgow’s  taxonomy,  Swanson,  Case  and  Vlerten 
(1991)  use  process  and  outcome  to  classify  evaluation  methods.  Their  examples  of  process  include  communication 
skills  in  tutorial  groups,  acceptance  of  responsibility  for  learning,  learning  to  learn,  use  of  learning  resources,  and 
development  of  problem-solving  skills,  while  they  use  outcome  assessment  to  indirectly  measure  the  quality  of  the 
learning  process  by  testing  the  results  of  that  process  after  it  has  been  operating  for  a period  of  time.  Barrow  and 
Tamblyn  (1980)  also  address  the  classification  of  process,  content,  and  outcome.  Bridge  and  Hallinger  (1995) 
classify  evaluation  methods  in  terms  of  who  structures  the  assessment  and  who  judges  the  performance.  According  to 
this  taxonomy,  four  evaluation  types  are  described  as  the  instructor-structured  and  student-judged  evaluation, 
instructor- structured  and  instructor-judged  evaluation,  student-structured  and  instructor-judged  evaluation,  and 
student-structured  and  student-judged  evaluation. 

In  describing  the  assessment  methods  I chose  process-oriented  and  outcome -oriented  approaches  to  classify 
them.  The  process  in  my  categorization  scheme  refers  to  students’  behaviors  and  the  outcome  to  their  products. 
According  to  Papham  (1990),  process  and  outcome  can  be  replaced  as  behavior  and  product,  and  this  two-part 
classification  scheme  takes  care  of  all  examinee  responses.  Unlike  products,  which  can  be  subsequently  measured, 
behaviors  disappear  once  they  occur.  So  we  must  make  sure  that  the  behavior  is  recorded.  Popham  emphasized  the 
importance  of  evaluating  students'  behaviors  and  notes  that  ’’sometimes  educators  will  wish  to  concentrate  on  the 
behaviors  used  by  a student  during  the  early  stages  of  the  student’s  creation  of  a product,”  that  is,  the  process.  On  the 
other  hand,  outcome-oriented  assessment  techniques  assess  a student’s  products,  such  as  essays  or  portfolios.  In  this 
study,  I regard  some  product  format  evaluation  tools,  for  example  videotape,  as  process-oriented  assessment  tools 
because  they  are  used  to  record  the  student’s  behavior.  Students’  behaviors  include  those  in  the  evaluation  process  or 
in  the  learning  process  that  are  evaluated. 

According  to  Reigeluth  (1983),  the  three  major  components  of  a theory  of  instruction  are  methods, 
conditions,  and  outcomes.  Methods  are  “the  different  ways  to  achieve  different  outcomes  under  different  conditions”. 
Conditions  are  defined  as  “factors  that  influence  the  effects  of  methods,”  while  outcomes  are  “the  various  effects  that 
provide  a measure  of  the  value  of  alternative  methods  under  different  conditions.”  In  this  section  I would  like  to 
apply  Reigeluth's  CMO  model  to  present  the  condition  variables,  the  process-oriented,  and  the  outcome-oriented 
assessment  methods  in  a problem-based  learning  approach.  Since  a detailed  description  of  learning  outcomes  is 
presented  earlier  in  this  paper,  the  intended  outcomes  of  each  assessment  method  will  be  mentioned  shortly  in  this 
section. 

Condition  variables  in  student  evaluation. 

The  condition  variables  of  student  assessment  fall  into  three  parts:  environment  / learner  characteristics, 
subject  content,  and  time  constraints.  Learner  characteristics  refer  to  the  attribute  each  learner  possesses,  such  as  the 
student’s  prior  experience  (e.g.  use  of  computers),  cognition  style,  and  psychosocial  traits,  which  will  influence  the 
learner’s  learning  and  interpersonal  skills.  Environment  variables  mean  the  availability  of  resource  materials, 
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computer  facilities,  and  classroom  capacity.  In  addition,  subject  content,  program  goals,  depth  of  problems  selected 
in  a PBL  course  (well-structured  vs.  ill-structured  problems),  and  available  time  are  important  variables  that  will 
influence  the  selection  of  evaluation  methods. 

Process-oriented  evaluation  methods 

1.  Tutor,  peer  ratings:  tutor  and  peer  ratings  are  common  assessment  techniques  that  can  be  used  to  assess 
communication,  group  cooperation,  effect,  and  self-directed  learning  skills  (Swanson,  Case,  and  Vleuten, 
1991).  Barrows  (1988)  describes  one  task  of  the  tutor  as  educational  diagnosis,  in  which  he  monitors  the 
progress  of  each  student  in  the  group  and  recognizes  any  learning  difficulties.  Techniques  of  peer  evaluation 
are  based  on  the  belief  that  group  members  are  in  a good  position  to  evaluate  each  other  during  the  learning 
process.  In  addition,  peer  evaluation  provides  a student  with  the  sense  of  control  over  the  behaviors  of  his 
group  members  before  he  is  confident  enough  to  enter  and  learn  wholeheartedly  in  the  group  (Allen,  Duch 
& Groh,  1996).  The  shortcoming  of  peer  evaluation  method  is  that  little  information  is  provided  because 
raters  can  only  give  an  overall  impression  of  co-workers  without  precise  ratings  of  distinct  skills.  The  skills 
to  be  assessed  are  listening  and  communication  skills,  interpersonal  skills,  group  cooperation,  self-directed 
learning  skills,  and  ownership. 

2.  Self-evaluation:  self-evaluation  refers  to  the  learners’  assessment  of  themselves,  how  they  review  their  own 
work,  knowledge,  or  skills  to  identify  inadequacies  (Barrows  and  Tamblyn,  1980).  Wilkerson  and  Feletti 
(1989)  suggest  using  PBL  to  encourage  students  to  assess  their  own  learning  through  discussion  in  small 
groups  to  compare  their  answers  with  those  of  others,  to  discover  errors  in  thinking,  and  to  hear  correct 
solutions.  Barrows  (1988)  also  suggests  students  in  a small  group  describing  how  they  did  in  terms  of 
reasoning  through  the  problem,  their  self-directed  learning  skills,  and  support  of  the  group  process.  Walton 
and  Matthews  (1989)  note  that  as  students  develop  the  self-directed  learning  skills,  their  self-assessment  and 
criticism  should  become  automatic.  Once  students  acquire  the  skills  of  reflection  and  deliberation,  they  will 
become  more  responsible  for  their  own  education.  In  terms  of  the  evaluation  tool,  Barrow  and  Tamblyn 
present  some  examples  to  help  students  assess  themselves  as  the  self-assessment  unit,  review  and 
comparison  of  criterion  performance,  peer  review,  and  review  with  faculty.  The  skills  to  be  assessed  are 
self-directed  learning  skills,  problem  solving  skills,  and  group-member  skills. 

3.  Unobtrusive  measure:  Swanson,  Case,  and  Vleuten  (1991)  list  some  usable  tools  that  can  be  used  to 
evaluate  the  student’s  learning  process.  They  are  library  records  of  books  and  articles  that  were  checked  out, 
computer  records  of  article  searches,  and  students’  logs  or  diaries  reviewing  their  learning  activities.  This 
method  is  better  for  curriculum  evaluation  than  student  assessment  because  it's  hard  to  standardize  all  the 
information  that  needs  to  be  assessed.  The  skills  to  be  assessed  are  skills  to  locate  and  use  useful  resources. 

4.  Oral  examination:  according  to  Barrows  and  Tamblyn  (1980),  the  oral  examination  ’’has  the  advantage  of 
establishing  a dialogue  or  interaction  between  the  student  and  the  examiner."  The  examiner  can  explore  the 
student’s  areas  of  strength  and  weakness  regarding  his  knowledge  and  thinking  ability.  The  examiner  can 
also  gain  a sense  about  the  student’s  self-evaluation,  personality,  and  his  reaction  under  pressure.  However, 
low  reliability  and  validity  may  be  a problem  due  to  personal  biases.  The  classic  example  is  the  ’’Triple 
jump  exercise’’  in  which  students  first  discuss  possible  problems  and  hypotheses  of  a given  problem  with 
the  examiner,  locate  and  review  resources,  and  then  discuss  the  solution  of  the  problem  with  the  oral 
examiner.  This  method  is  used  at  McMaster  University  to  assess  problem  solving  and  self-directed  learning 
(Norman,  1991).  The  skills  to  be  assessed  are  problem-solving  skills,  self-directed  learning  skills,  and 
knowledge  of  the  content  area. 

5.  Observation  of  a patient  interview  and  examination:  the  student's  interview  of  a real  or  simulated  patient 
is  observed  by  the  teacher.  The  observation  allows  the  teacher  to  estimate  the  student’s  abilities  to  choose 
appropriate  questions,  his  interview,  inquiry  and  physical  examination  skills,  and  use  of  time.  The 
weaknesses  of  this  method  are  non-standard  criteria  of  student  performance  and  the  bias  of  examiners.  In 
addition,  patients  vary  in  a wide  range  and  cause  uncontrolled  variables. 

6.  Patient  problem  simulations:  Barrows  and  Tamblyn  (1980)  discuss  patient  problem  simulations  as  an 
evaluation  method  that  is  based  on  the  simulation  of  the  patient  problem.  Simulated  situations  are  provided 
in  several  formats  for  students  to  react  to  and  interact  with.  Students'  behaviors  are  recorded  or  observed  for 
evaluation.  Examples  of  this  are  scenarios  that  deal  with  sequential  management  problems,  portable  patient 
problem  packs,  simulated  patients,  mechanical  mannequins,  and  use  of  models. 
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(1)  Sequential  management  problems:  an  opening  statement  about  the  patient  is  presented  to  a student  so  that  he  can  to 
describe  the  possible  problems  of  the  patient  and  the  actions  to  take  next.  Skills  assessed  include  the  perception  and 
interpretation  of  data,  problem  formulation,  generation  of  hypotheses,  and  inquiry  strategies. 

(2)  Portable  patient  problem  pack:  using  this  method  represents  a patient  problem  in  playing-card  format  that  gives  a 
student  the  opening  picture  of  the  patient’s  problem.  The  student  can  choose  any  actions  in  any  sequence.  After  he  chooses 
an  action,  he  can  get  correct  information  by  reading  the  back  of  the  card.  His  inquiry  path  is  recorded  by  the  sequence  of 
card  he  chooses.  Skills  assessed  include  cognitive  process,  data  perception  and  interpretation,  hypothesis  generation, 
problem  formulation,  inquiry  strategy,  and  diagnostic  decision-making. 

(3)  Simulated  patients:  this  method  is  the  same  as  "observation  of  student  interview  and  examination”  mentioned  before. 

(4)  Mechanical  mannequins:  the  assessment  is  based  on  students’  observation  or  evaluation  of  mechanical  simulations  of  the 
human  body  or  portions  of  the  human  body.  It  is  an  excellent  way  for  complex  medical  skills  to  be  learned  at  no  risk  to 
patients.  Computers  are  used  to  simulate  patient  symptoms  and  to  record  the  various  actions  of  students.  At  the  end  of  the 
exercise,  the  computer  will  print  out  the  record  of  performance,  time  used,  dosage  of  medicines,  and  other  steps.  Skills 
evaluated  include  the  execution  of  complex  skills  in  observation,  physical  examination,  and  medical  intervention. 

(5)  Models:  students  manipulate  and  evaluate  mock-ups  of  the  various  parts  of  the  body  represented  with  plastic,  plaster, 
metal,  and  wood  dummies.  Motor  skills  are  evaluated. 

7.  Review  of  case  record:  review  of  case  record  is  a technique  that  belongs  to  both  process-oriented  and 
outcome-oriented  assessment  methods.  Teachers  directly  score  students'  case  records.  Case  reports  and 
patient’s  reports  are  used  as  evaluation  tools.  According  to  Barrows  and  Tamblyn  (1980),  the  case  report 
provides  information  that  cannot  be  evaluated  during  the  encounter;  the  patient's  report  evaluates  student’s 
ability  to  communicate  in  writing.  Audit  record  is  another  tool  used  to  collect  information  about  what  a 
student  really  did  in  his  work  with  patients.  The  skills  to  be  assessed  are  the  ability  to  recognize  important 
data,  interpret  that  data,  formulate  hypotheses,  make  decisions,  write  diagnoses,  and  give  treatment  (The 
last  three  skills  are  products). 

8.  Authentic  assessment:  Herman,  Aschbacher,  and  Winters  (1992)  note  the  characteristics  of  authentic 
assessment  as  relevance  and  meaningfulness  to  students.  It  uses  contextualized  problems  to  evaluate 
students'  complex  skills.  There  is  no  single  correct  answer  in  the  authentic  assessment.  Since  students  know 
public  standards  in  advance  and  the  evaluation  focus  on  individual  pacing  and  growth,  it  creates  a fear  free 
evaluation  environment.  Authentic  assessment  should  be  a part  of  the  learning  process,  for  Barrows  (1994) 
regards  authentic  assessment  as  a natural  component  of  authentic  learning. 

9.  Performance  assessment:  according  to  Barrows  and  Myers  (1993),  a wide  variety  of  performances  that 
can  be  evaluated  in  a PBL  program  include  oral  presentations,  written  presentations,  audiovisual 
presentations,  dramatic  presentations,  other  fine  arts  presentations,  illustrations,  graphs,  mathematical 
analyses,  and  portfolios.  Students  demonstrate  their  knowledge  as  they  report  their  thinking  and  problem- 
solving process.  The  skills  to  be  assessed  are  creativity  and  communication  skills. 

Outcome-oriented  evaluation  methods 

1.  Review  of  case  record:  see  assessment  method  number  7 in  the  process-oriented  evaluation  methods. 

2.  Student-judged  evaluation:  in  Bridge  and  Hallinger’s  (1995)  book,  they  present  several  evaluation 
methods  applied  in  a project-based  curriculum.  The  five  student-judged  evaluation  techniques  are 
integrative  essay,  protocols,  models  or  examples,  knowledge  review  exercise,  and  probe  questions. 

( 1 ) Integrative  essay:  Bridges  and  Hallinger  describe  a problem-based  curriculum  for  which  the  basic  unit  of  instruction  is  a 
project  and  students  use  class  time  to  meet  in  the  project  teams.  Students  prepare  an  integrative  essay  following  each 
project.  The  structures  of  this  integrative  essay  are  based  on  two  procedures:  students  discuss  what  they  have  learned  and 
how  they  might  use  the  knowledge;  a list  of  questions  is  given  for  the  students  to  choose  and  discuss. 

(2)  Protocols:  the  elements  in  a protocol  include  instructions,  guidelines,  or  checklists  for  students  to  guide  their 
performance.  In  the  example  mentioned  by  the  authors,  students  use  protocols  to  judge  their  memos  that  are  written  during 
the  learning  process. 

(3)  Models  or  examples:  students  produce  products  in  the  project-based  curriculum.  Models  or  examples  are  instances  of 
products  provided  by  teachers  for  students  to  contrast  with  their  own  products  for  self-evaluation. 

(4)  Knowledge-review  exercises:  knowledge-review  exercises  are  distributed  at  the  beginning  of  each  project.  Students  may 
use  them  as  pretests  or  posttests  or  both. 

(5)  Probe  questions:  the  skills  to  be  assessed  are  A set  of  key  questions  is  given  to  students  at  the  conclusion  of  some 
projects  that  stimulate  their  thinking  about  their  final  products  or  performances. 

3.  Multiple-choice  examinations:  this  format  has  been  developed  into  a sophisticated  method  to  assess  recall 
of  content  knowledge,  intellectual  skills,  or  attitudinal  dispositions.  It  has  the  advantage  of  presenting  a 
large  number  of  questions,  which  can  be  answered  by  students  in  a short  time.  However,  Swanson,  Case, 
and  Vleuten  (1991)  claim  that  this  method  is  often  rejected  in  a PBL  program  because  it  can  only  assess 
factual  knowledge.  This  format  is  argued  by  some  educators  to  provide  cues  to  the  examinees  and  leads  to 
suggestions  of  right  answers. 

4.  Short-answer  or  word-completion  examinations:  this  method  is  particularly  suitable  for  measuring 
simple  types  of  learning  outcomes,  like  recall  of  factual  information.  It  has  the  advantage  of  not  cueing  the 
student  to  the  right  answer.  But  it  does  not  evaluate  inquiry  strategies  or  interpersonal  skills. 
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5.  Essay  examinations:  according  to  Swanson,  Case,  and  Vleuten  (1991),  this  format  can  provide  an  in-depth 
assessment  of  problem-solving  skills  and  encourage  students  to  integrate  their  insight  into  a systematic 
framework.  In  addition,  students'  creativity  and  the  abilities  of  organizing  and  expressing  ideas  can  be 
evaluated.  The  essay  question  can  be  used  to  ask  students  to  justify  their  decisions  or  explain  relevant 
underlying  principles.  With  good  design  of  exam  questions,  this  method  can  evaluate  students'  abilities  to 
use  knowledge  in  solving  important  problems.  It  can  measure  complex  as  well  as  simple  types  of  learning 
outcomes.  But  it  has  the  problem  of  poor  general izability  of  scores,  which  its  reliability  somewhat 
problematic. 

6.  Patient  problem  simulations 

(1)  Audiovisual  examination:  there  are  a variety  of  ways  that  audiovisual  media  are  used  in  place  of  clinical  skill  exams.  It 
can  measure  students'  abilities  to  make  observations  and  interpret  data  but  it  cannot  be  used  to  evaluate  other  important 
clinical  competencies,  such  as  inquiry  strategies  and  interpersonal  skills  (Barrows  & Tamblyn,  1980). 

(2)  Compute r-based  clinical  simulations:  in  this  simulation,  the  student  is  presented  with  an  "opening  scenario"  that 
provides  a description  of  a patient  problem  and  then  is  given  a set  of  options  to  choose  from  (Swanson,  Case,  &.  Vleuten, 
1991).  Additional  information  is  gathered  when  the  student  proceeds  through  a series  of  scenes.  The  computer  challenges 
the  student’s  data  perception  and  interpretation  skills,  the  ability  to  intervene  with  a patient  problem,  and  inquiry  strategy. 

7.  Portfolio  assessment:  Herman,  Aschbacher,  and  Winters  (1992)  describe  recent  trends  in  assessment  and 
suggest  using  portfolio  review  as  the  basis  for  assessment  by  the  teacher,  students,  and  parents.  They  define 
portfolios  as  "collections  of  student  work  that  are  reviewed  against  criteria  in  order  to  judge  an  individual 
student  or  a program."  The  types  of  student  works  could  be  essays,  videotapes,  art,  journal  entries,  and  so 
on.  In  order  to  evaluate  students  using  portfolio  assessment,  the  teacher  should  define  the  assessment 
purpose,  set  methods  for  determining  what  to  include  in  the  portfolio  by  whom  and  when,  and  define  the 
criteria  for  sample  selection  and  judgment.  Two  skills  in  the  example  proposed  by  the  authors  are  problem- 
solving and  communication  skills. 

Conclusion 

By  reviewing  articles  in  student  assessment,  one  notices  that  assessment  methods  have  changed,  with  their 
underpinning  epistemology  shifting  from  behaviorism  to  cognitivism  to  constructivism.  There  are  more  emphases  on 
the  importance  of  self-reflection  and  assessment  in  the  learning  context  in  recent  years.  Different  evaluation  methods 
are  applied  based  on  these  epistemologies  to  better  capture  significant  and  enduring  learning  outcomes.  Many 
evaluation  methods  described  in  this  paper  are  applied  in  medical  schools  to  assess  students’  clinical  abilities.  On  the 
other  hand,  some  techniques  are  used  to  evaluate  students’  problem-solving  skills  and  various  competencies  in 
different  fields.  Techniques  used  for  process-oriented  evaluation  have  beneficial  effects  on  student  learning,  while 
outcome-evaluation  techniques  have  the  advantages  of  ease  of  scoring  and  conducting.  When  making  decisions  in 
choosing  appropriate  evaluation  methods,  it  is  important  to  take  the  condition  variables  and  desired  outcome 
variables  into  consideration. 
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"Theory  helps  us  bear  our  ignorance  of  fact" 

G.  Santayana 1896 

In  the  rapidly  changing  and  diverse  environment  in  which  distance  education  is  practiced,  many  questions 
remain  unanswered.  In  this  environment  it  is  difficult  to  arrive  at  one  approach  to  guide  the  practice  of  distance 
education.  New  technologies  and  new  ideas  about  student  learning  challenge  the  traditional  techniques  for  the 
practice  of  distance  education.  This  theme  of  change  is  evident  in  the  discussions  of  distance  education  and  its 
definitions  and  theories. 

Numerous  definitions  for  distance  education  have  been  proposed,  and  most  include  the  separation  of 
teacher  and  learner,  the  influence  of  an  educational  organization,  the  use  of  media  to  unite  teacher  and  learner,  the 
necessity  for  two-way  communication,  and  the  potential  for  individualized  learning.  The  classical,  European 
definitions  of  distance  education  concentrate  on  teaching  and  learning  taking  place  at  different  times  and  in  different 
places,  while  recent  definitions  enabled  by  new  interactive  technologies  emphasize  educational  experiences 
occurring  at  the  same  time  in  different  locations. 

Many  cringe  at  the  thought  of  a discussion  of  theory.  This  need  not  be  the  case.  Theory  is  important  to  the 
study  of  distance  education  because  it  directly  affects  the  practice  of  the  field.  Traditionally,  theories  of  distance 
education  have  come  from  sources  external  to  the  USA.  Recently,  the  field  in  the  United  States  has  matured  to  the 
point  where  indigenous  definitions  and  theories  have  begun  to  emerge;  most  notably  theories  based  on  the  American 
system  of  education.  Local  control  of  the  curriculum,  elected  school  governing  boards,  small  classes,  rapport 
between  teacher  and  students,  community-based  funding,  and  highly  personalized  instruction  are  important 
characteristics  of  the  American  system  of  education.  Many  educators  are  now  advocating  the  need  for  forms  of 
distance  education  that  retain  traditional  aspects  of  American  education  while  taking  advantage  of  the  opportunities 
of  telecommunications  systems.  In  this  environment,  equivalency  theory  has  emerged. 

The  Need  for  Theory 

Keegan  (1986)  reaffirmed  the  continued  need  for  a theory  of  distance  education  when  he  said: 

Lack  of  accepted  theory  has  weakened  distance  education;  there  has  been  a lack  of  identity,  a sense  of  belonging  to 
the  periphery  and  the  lack  of  a touchstone  against  which  decisions  on  methods,  on  media,  on  financing,  on  student 
support,  when  they  have  to  be  made,  can  be  made 
with  confidence,  (p.  63) 

Keegan  (1995)  has  noted  that  a theory  is  something  that  eventually  can  be  reduced  to  a phrase,  a sentence  or 
a paragraph,  and  which  while  subsuming  the  practical  research,  gives  the  foundation  on  which  the  structures  of  need, 
purpose,  and  administration  can  be  erected. 

Holmberg  (1985)  defined  theory  as: 

A systematic  ordering  of  ideas  about  the  phenomenon  of  a field  of  inquiry,  and  an  overarching  logical  structure  of 
reasoned  suppositions  which  can  generate  testable  hypotheses  (p.  4). 

Holmberg  suggested  that  distance  education  has  been  characterized  by  a trial  and  error  approach  with  little 
consideration  being  given  to  a theoretical  basis  for  decision-making,  and  that  the  theoretical  underpinnings  of 
distance  education  are  fragile.  Most  efforts  in  this  field  have  been  practical  or  mechanical  and  have  concentrated  on 
the  logistics  of  the  enterprise. 

In  his  landmark  work,  The  Foundations  of  Distance  Education , Keegan  classified  theories  of  distance 
education  into  three  groups:  theories  of  independence  and  autonomy,  theory  of  industrialization  of  teaching,  and  the 
theory  of  interaction  and  communication. 

Theories  of  Independent  Study 
#7  - 'Wedemeyer 

For  Wedemeyer  (Keegan,  1986),  the  essence  of  distance  education  was  the  independence  of  the  student. 

This  was  reflected  in  Wedemeyer's  preference  for  the  term  “independent  study”  for  distance  education  at  the  college 
or  university  level.  Wedemeyer  was  critical  of  contemporary  patterns  of  higher  education  and  believed  that  outdated 
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concepts  of  learning  and  teaching  were  being  employed.  It  was  felt  that  universities  failed  to  utilize  modem 
technologies  in  ways  that  could  alter  an  institution. 

Wedemeyer  set  forth  a system  with  characteristics  emphasizing  learner  independence  and  adoption  of 
technology  as  a way  to  implement  that  independence.  Wedemeyer  noted  four  elements  of  every  teaching-learning 
situation:  a teacher,  a learner  or  learners,  a communications  system  or  mode,  and  something  to  be  taught  or  learned. 
He  proposed  a reorganization  of  these  elements  that  would  accommodate  physical  space  and  allow  greater  learner 
freedom.  Key  to  the  success  of  distance  education,  Wedemeyer  believed,  was  the  development  of  the  relationship 
between  student  and  teacher. 

#2  - Moore 

Formulated  in  the  early  1970s,  Moore’s  theory  of  distance  education  was  a classification  method  for 
distance  education  programs.  It  examined  two  variables  in  educational  programs:  the  amount  of  learner  autonomy 
and  the  distance  between  teacher  and  learner. 

For  Moore  (1994),  distance  is  composed  of  two  elements,  each  of  which  can  be  measured.  First  is  the 
provision  for  two-way  communication  (dialog).  Some  systems  or  programs  offer  greater  amounts  of  two-way 
communication  than  others.  Second  is  the  extent  to  which  a program  is  responsive  to  the  needs  of  the  individual 
learner  (structure).  Some  programs  are  very  structured,  while  others  are  very  responsive  to  the  needs  and  goals  of  the 
individual  student. 

In  the  second  part  of  his  theory,  Moore  addressed  learner  autonomy.  He  notes  that  in  traditional  school 
settings  learners  are  very  dependent  on  teachers  for  guidance,  and  that  in  most  programs,  conventional  and  distance, 
the  teacher  is  active,  while  the  student  is  passive. 

In  distance  education  there  is  a gap  between  teacher  and  student,  so  the  student  must  accept  a high  degree  of 
responsibility  for  the  conduct  of  the  learning  program.  The  autonomous  learner  needs  little  help  from  the  teacher, 
who  may  be  more  of  a respondent  than  a director.  Some  adult  learners,  however,  require  help  in  formulating  their 
learning  objectives  and  in  identifying  sources  of  information  and  in  measuring  objectives. 

Theory  of  Industrialization  of  Teaching 

After  examining  a research  base  that  included  an  extensive  analysis  of  the  distance  teaching  organizations 
of  the  1960s,  Peters  (1988)  proposed  that  distance  education  could  be  understood  and  practiced  by  comparison  to  the 
industrial  production  of  goods.  Peters  stated  that  from  many  points  of  view  conventional,  oral,  group-based 
education  was  a pre- industrial  form  of  education.  Distance  education,  on  the  other  hand,  permits  the  incorporation  of 
characteristics  of  industry,  which  makes  education  standardized,  widely  available,  and  extremely  cost  effective.  His 
statements  implied  that  distance  teaching  could  not  have  existed  before  the  industrial  era.  Critical  to  this  theory  are 
concepts  such  as: 


• Division  of  labor  - which  means  that  different  professionals  and  paraprofessionals  have  specific  roles  in  the  teaching  and 
learning  process. 

• Mechanization  - which  means  that  machines  and  devices  are  a required  component  of  the  teaching  process. 

• Assembly  Line  - which  implies  that  techniques  are  used  so  materials  and  students  move  past  or  are  involved  at  different 
times  with  distance  education  professionals  who  have  specific  duties. 

• Mass  Production  - which  means  that  techniques  are  used  to  develop  large  quantities  of  instructional  materials  very 
inexpensively  in  order  to  reach  large  numbers  of  learners  uniformly. 

• Standardization  - which  means  the  entire  instructional  process  is  made  standard  according  to  pre-established  criteria  so 
that  learners  receive  the  same  curriculum  in  uniform  ways. 

• Concentration  and  Centralization  - this  means  that  a monopoly  is  instituted  so  that  one  organization  provides  educational 
experiences  to  large  numbers  of  students. 

Theory  of  Interaction  and  Communication 

Holmberg’s  (1989)  theory  of  distance  education,  which  he  calls  guided  didactic  conversation,  falls  into  the 
general  category  of  communication  theory.  Holmberg  noted  that  his  theory  had  explanatory  value  in  relating  teaching 
effectiveness  to  the  impact  of  feelings  of  belonging  and  cooperation  as  well  as  to  the  actual  exchange  of  questions, 
answers,  and  arguments  in  distance  education. 

Distance  teaching  will  be  successful  to  the  extent  it  supports  student  motivation,  promote  learning  pleasure  and 
make  study  relevant  to  the  individual  learner  and  his/her  needs,  creating  feelings  of  rapport  between  the  learner  and 
the  distance-education  institution  (its  tutors,  counselors,  etc.),  facilitating  access  to  course  content,  engaging  the 
learner  in  activities,  discussions  and  decisions  and  generally  catering  for  helpful  real  and  simulated  communication 
to  and  from  the  learner. 

Holmberg  himself  noted  that  this  was  admittedly  a leaky  theory.  However,  he  added,  it  was  not  devoid  of 
explanatory  power.  It  did,  in  fact,  indicate  essential  characteristics  of  effective  distance  education. 
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Equivalency  Theory  - An  Emerging  Approach  to  Distance  Education 

New  telecommunications  technologies  have  impacted  significantly  on  education.  Many  feel  that  the 
availability  of  powerful  telecommunications  systems  is  responsible  for  the  popularity  of  distance  education  in  the 
USA.  Interactive  television  systems,  such  as  the  Iowa  Communications  Network  (Simonson  & Schlosser,  1995), 
permit  distant  learners  and  instructors  to  see  and  be  seen,  hear  and  be  heard,  and  learn  and  teach  in  almost  the  same 
manner  as  they  would  if  everyone  was  in  the  local  classroom. 

Keegan  (1995)  suggested  that  electronically  linking  instructor  and  students  at  various  locations  could  create 
a virtual  classroom. 

The  theoretical  analyses  of  virtual  education,  however,  have  not  yet  been  addressed  by  the  literature:  Is  virtual 
education  (interactive,  live  televised  instruction)  a subset  of  distance  education  or  to  be  regarded  as  a separate  field 
of  educational  endeavor?  What  are  its  didactic  structures?  What  is  the  relationship  of  its  cost  effectiveness  and  of 
its  educational  effectiveness  to  distance  education,  and  to  conventional  education? 

It  is  in  this  environment  of  virtual  education  that  the  equivalency  theory  of  distance  education  has  emerged. 
Equivalency  Theory  - 

Distance  education's  appropriate  application  should  provide  equivalent  learning  experiences  for  all  students  distant 
and  local  - in  order  for  there  to  be  expectations  of  equivalent  outcomes  of  the  educational  experience. 

Stated  another  way,  this  theory  is  based  on  research  that  indicates  that  learning  at  a distance  and  learning 
locally  are  fundamentally  different,  even  when  interactive  technologies  are  used.  Equivalency  theory  advocates  the 
design  for  distant  and  local  learners  of  a collection  of  probably  different  but  ultimately  equivalent  learning 
experiences.  The  objective  of  the  designer  of  distance  education  instruction  is  to  provide  for  appropriate  learning 
experiences  for  each  student  that  are  based  on  their  unique  needs,  including  physical  location. 

This  theory  is  based  on  this  definition  of  distance  education. 

Distance  education  is  a formal  and  institutionally-based  educational  system  where  learners  and  teachers  are 
separated  from  one  another,  and  where  voice,  video,  and  data  interaction  occurs  using  telecommunication  systems 
(Simonson  & Schlosser,  1995). 

Instructional  experiences  are  essential  to  learning.  It  should  not  be  necessary  for  any  group  of  learners,  local 
or  distant,  to  compensate  for  different,  possibly  lesser,  instructional  experiences.  Students  should  have  learning 
experiences  designed  and  made  available  to  them  that  are  tailored  for  the  environment  and  situation  in  which  they 
find  themselves.  Thus,  those  developing  distance  education  systems  should  strive  to  provide  appropriate  learning 
experiences  for  students  no  matter  how  they  are  linked  to  the  resources  or  instruction  they  require. 

One  key  to  this  theory  is  the  concept  of  equivalency.  Local  and  distant  learners  have  fundamentally  different 
environments  in  which  they  learn.  It  is  the  responsibility  of  the  distance  educator  to  design  instructional  events  that 
provide  learning  experiences  for  individuals  and  groups  of  students.  A triangle  and  a square  that  have  the  same  area 
are  considered  equivalent  even  though  they  are  quite  different  geometrical  shapes.  Similarly,  the  experiences  of  the 
local  learner  and  the  distant  learner  should  have  equivalent  value  even  though  specific  experiences  might  be  quite 
different. 

Also  a key  to  this  approach  is  the  concept  of  learning  experience.  A learning  experience  is  anything  that 
happens  to  or  with  the  student  that  promotes  learning,  including  what  is  observed,  felt,  heard,  or  done.  It  is  likely  that 
students  in  various  locations,  learning  at  different  times,  may  require  a different  mix  of  learning  experiences.  Some 
will  need  a greater  amount  of  observing,  and  others  a larger  dosage  of  doing.  The  goal  of  instructional  planning  for 
distance  education  is  to  develop  an  approach  that  makes  the  sum  of  experiences  for  each  learner  equivalent. 
Instructional  design  procedures  should  attempt  to  anticipate  and  provide  the  collection  of  experiences  that  will  be 
most  suitable  for  each  student  or  group  of  students. 

A final  key  to  equivalency  theory  is  the  idea  of  telecommunications,  which  means  communicating  at  a 
distance.  Most  often  in  the  USA,  telecommunications  refers  to  electronic  telecommunications  systems  such  as  those 
using  synchronous  audio,  video,  and  computer  networks.  Synchronous  interaction  need  not  be  the  mandatory, 
however.  Various  communications  systems,  including  asynchronous  ones,  can  and  should  be  used  for  distance 
education,  as  long  as  the  goal  of  equivalency  of  experiences  is  met. 

The  equivalency  approach  is  based  on  core  values  such  as  local  control  and  personalized  instruction  that  are 
held  almost  sacred  in  classical  American  education.  If  distance  education  is  to  be  widely  accepted  and  routinely 
available  it  must  be  high  quality,  easily  obtained,  and  familiar  to  those  in  need.  Powerful,  modem 
telecommunications  systems  enable  the  delivery  of  powerful  and  recognizable  learning  experiences  to  students 
anytime  and  any  place. 
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Summary 

The  changing  and  diverse  environment  in  which  distance  education  is  practiced  has  inhibited  the 
development  of  a single  theory  upon  which  to  base  practice  and  research.  A variety  of  theories  have  been  proposed 
to  describe  classical  distance  education.  They  include  theories  that  emphasize  independence  and  autonomy  of  the 
learner,  industrialization  of  teaching,  and  interaction  and  communication.  These  traditional  theories  emphasize  that 
distance  education  is  a fundamentally  different  form  of  education. 

Recent  emerging  theories  based  on  the  capabilities  of  new  interactive  audio,  video,  and  computer  systems, 
state  that  distance  education  is  not  a distinct  field.  These  approaches  utilize  existing  educational  theory  and  advocate 
the  creation  of  equivalent  experiences  for  distant  and  local  learners. 

Theory  helps  guide  the  practice  of  a field,  and  permits  a certain  “ignorance  of  fact.”  Equivalency  theory 
provides  a framework  for  design  and  production  of  instructional  experience  for  local  and  distant  learners  that  need 
not  be  the  same.  The  key  will  be  whether  equivalent  experiences  produce  equivalent  outcomes. 
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Best  of  Both  Worlds  :The  Nova  ITDE  Model  of  Distance  Education 
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Nova  Southeastern  University 


Friday,  8 p.m. 


One  by  one,  the  students’  faces  appear  on  the  computer  screen,  and  with  broad  smiles  greet  their 
classmates  in  turn.  A class  of  15,  they  are  participating  in  the  course,  Future  Trends:  Beyond  Emerging 
Technology,  from  their  homes— from  California  to  New  York  and  from  Washington  to  Florida.  The 
students  know  each  other  well,  having  taken  classes  together  for  more  than  two  years.  On  the  computer 
screen,  the  wide-angle  view  and  the  somewhat  unsteady  motion  indicates  that  compressed  video  is  being 
used  to  link  the  students.  But  the  image  is  in  color  and  the  quality  is  surprisingly  good L 
Two  others  join  the  videoconference:  a program  professor  greets  the  students  from  a studio  on  the  campus 
of  Nova  Southeastern  University  (NSU),  in  suburban  Fort  Lauderdale  and  explains  that  the  course 
instructor  has  been  delayed,  but  will  join  the  session  shortly.  The  other  participant  in  the  videoconference 
is  the  class’  cluster  coordinator  from  her  home  in  Florida.  Lest  there  be  any  doubt  about  her  location,  a 
huge  banner,  reading  Hollywood,  is  visible  on  the  wall  behind  her . 

Within  a few  minutes,  the  course’s  instructor,  an  adjunct  professor,  joins  the  videoconference,  greets  the 
participants,  and  apologizes  for  the  delay.  The  videoconferencing  equipment  is  straightforward,  but 
connections  from  his  office,  in  Rio  de  Janeiro,  are  problematic.  For  several  minutes,  the  instructor 
discusses  the  goals  of  the  evening  and  fields  questions  from  the  students  about  course  readings  and 
assignments. 

In  the  following  two  hours,  the  students  make  short  presentations  based  on  assigned  readings , As  each 
student  presents,  classmates  follow  along  with  the  help  of  handouts  that  had  been  distributed  via  email. 
The  students  use  a variety  of  presentation  media:  there  are  murmurs  of  approval  as  one  student  uses  his 
camcorder  as  a document  camera,  and  envious  hoots  as  another  student  couples  a PowerPoint 
presentation  with  a live  picture-in-picture  image  of  himself 

As  the  session  ends,  the  instructor  discusses  the  agenda  for  upcoming  videoconferences  as  well  as  for  the 
weeklong  summer  institute  to  be  held  in  Fort  Lauderdale,  where  the  group  will  gather  for  intensive 
instruction  the  following  month. 


The  scenario  “Friday,  8 p.m.”  is  drawn  from  the  experiences  of  Cluster  One,  whose  members  became,  in 
February  of  1996,  the  First  group  of  students  enrolled  in  Nova  Southeastern  University’s  program  in  Instructional 
Technology  and  Distance  Education  (ITDE).  For  this  cluster,  use  of  desktop  video  was  an  experiment — one  of  many 
they  had  experienced  in  their  graduate  program.  Over  the  course  of  three  years,  they  have  been,  in  the  words  of  a 
professor,  “put  through  the  wringer.”  But  in  their  use  of  a wide  variety  of  communications  technologies,  the  students 
gained  a rare  and  valuable  opportunity — hands-on  experience  coupled  with  a deeper  understanding  of  these 
technologies  and  their  use  in  distance  education.  Now,  after  three  years,  these  students  are  poised  to  receive  their 
Doctor  of  Education  degrees.  It  seems  appropriate  at  this  time  to  examine  to  examine  the  “Nova  ITDE  Model,”  the 
distinctive  multi-mode  approach  to  distance  education  which  the  students  of  Cluster  One  experienced  and  helped 
refine. 

The  ITDE  Program 

The  ITDE  program  was  established  in  the  spring  of  1996  with  a single  “cluster” — or  cohort — of  27.  Seven 
of  the  students  were  enrolled  in  the  Master  of  Science  program,  the  remaining  20  in  the  Ed.D.  program.  Three  years 
later,  ITDE  has  12  clusters,  a total  of  about  250  students,  all  of  whom  meet  in  Fort  Lauderdale  three  times  each  year. 
Students  in  these  clusters  are  mostly  from  the  United  States  with  a sprinkling  of  students  from  other  countries, 
including  Israel,  Germany,  and  the  Netherlands.  ITDE  is  also  fostering  the  development  of  two  “international” 
clusters — one  in  Jamaica,  the  other  in  Venezuela,  whose  students  will  receive  the  bulk  of  their  instruction  in  their 
home  countries. 

Nova  Southeastern  University 

Founded  in  1964,  Nova  Southeastern  University  (NSU)  is  the  largest  private  university  in  Florida,  with  an 
enrollment  of  about  15,000.  Based  in  suburban  Fort  Lauderdale,  it  offers  undergraduate  and  graduate  programs 
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leading  to  degrees  in  education,  law,  psychology,  oceanography,  osteopathic  medicine,  dental  medicine,  pharmacy, 
and  many  other  fields. 

Fischler  Graduate  School  of  Education  and  Human  Services 

From  its  18-acre  North  Miami  Beach  campus,  the  Fischler  Graduate  School  offers  courses  in  six  program 
areas,  leading  to  the  master’s,  educational  specialist,  and  doctoral  degrees.  Approximately  8,000  students  are 
enrolled  in  six  programs:  Graduate  Teacher  Education,  Life  Span  Care  and  Administration,  Educational  Leaders, 
Higher  Education,  Communication  and  Science  Disorders,  and  Education  and  Technology. 

The  Nova  ITDE  Model  of  Distance  Education 

The  Nova  ITDE  model  of  distance  education  is  characterized  by  a unique  combination  of  pedagogical, 
structural,  and  technological  attributes.  The  most  important  of  these  attributes  are:  a uniform  course  of  study 
offering  a single  major,  the  cluster  concept,  the  applied  thesis  and  dissertation,  and  face-to-face  instruction  combined 
with  the  use  of  a wide  variety  of  distance  education  technologies. 

The  Course  of  Study 

Over  the  course  of  their  21 -month  program,  master’s  students  in  the  ITDE  program  participate  in  four  study 
areas,  each  of  which  is  composed  of  two  courses.  The  study  areas,  in  the  order  taken,  are: 

• Leadership  I:  Influence  of  Technology  on  Leadership  and  Management 

• Research  and  Evaluation  I 

• Summer  Institute  I:  Instructional  Media 

• Instructional  Design 

Doctoral  students  take  the  same  courses,  plus  the  following  study  areas: 

• Research  and  Evaluation  II 

• Summer  Institute  0:  Instructional  Systems 

• Management  and  Human  Resources  Development 

• Summer  Institute  HI:  Trends 

• Leadership  D:  Leadership,  Technology,  and  Power 

The  Cluster  Concept 

Since  the  founding  of  NSU  in  1964,  the  cluster  concept  has  been  a strength  of  the  university’s  graduate 
education  programs.  Each  cluster  in  the  ITDE  program  is  a group  of  20-30  students  who  live  and  work  in  a wide 
variety  of  geographic  locations  and  professional  settings.  Each  cluster  operates  under  the  direction  of  a cluster 
coordinator.  The  coordinator,  who  holds  a doctorate  in  education,  technology,  or  a related  field  from  NSU,  is  a 
facilitator  of  numerous  administrative  details  and  cluster  activities  and  serves  to  support  and  advise  students.  The 
coordinator  monitors  student  progress,  offers  academic  advice  and  encouragement,  and  facilitates  communication 
that  fosters  interaction  among  students  who  have  limited  face-to-face  contact. 

The  cluster  coordinator  is  required  to  be  readily  accessible  via  email,  to  maintain  the  cluster  Web  page,  to 
publish  Web-based  newsletters,  and  attend  online  class  sessions.  In  this  unique  position,  the  coordinator  is  a support 
person  for  the  students,  a facilitator  for  senior  faculty  and  guest  lecturers,  and  a local  site  administrator/manager  for 
program  office  staff. 

The  cluster  is  the  administrative  unit  that  links  students  throughout  the  program.  Strong  bonds  develop 
between  the  student  and  the  coordinator.  Friendship,  collegiality,  academic  support  and  encouragement  develop  as 
students  progress  through  a series  of  study  areas.  Student  program  evaluations  consistently  indicate  that  the  cluster 
concept  is  of  extremely  high  value.  The  leadership  of  the  coordinator  is  considered  a crucial  factor  for  student 
success. 

The  Instruction 

ITDE  students  enroll  in  three  four-month  study  areas  per  year,  in  which  most  content  is  delivered  through 
readings  and  face-to-face  instruction.  Discussions,  student  presentations,  and  interaction  with  instructors  are 
conducted  at  a distance.  Each  study  area,  led  by  an  NSU  program  professor  or  an  adjunct  supervised  by  a program 
professor,  consist  of  three  distinct  phases. 

In  the  first  phase,  students  are  assigned  extensive  readings,  usually  including  several  texts  and  a collection 
of  journal  articles,  in  addition  to  the  study  guide  that  is  a component  of  all  ITDE  courses.  During  this  portion  of  the 
course,  students  one  or  more  audioconferences,  in  which  introductions  are  made  and  the  instructor  discusses  goals 
and  objectives  of  the  study  area,  fields  questions,  and  perhaps  provides  a modest  amount  of  instruction.  Finally, 
students  complete  a significant  assignment  based  on  the  readings. 
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The  second  phase  of  the  study  area,  which  takes  place  in  Fort  Lauderdale,  is  devoted  to  intensive  face-to- 
face  instruction  during  an  extended  weekend  (three  or  four  days)  or  an  extended  weeklong  summer  institute.  If  more 
than  one  cluster  is  taking  the  course  at  any  given  time  (as  is  increasingly  the  case),  the  clusters  are  blended.  The 
instructors  (one  per  cluster)  work  with  each  of  the  students.  This  arrangement  allows  students  to  meet  and  work  with 
other  students  in  the  program,  as  well  as  to  experience  a variety  of  teaching  styles  and  establish  connections  with 
nationally- known  instructors.  Before  the  end  of  the  second  phase  of  the  study  area,  learning  is  assessed,  and  students 
begin  work  on  a major  final  project. 

During  the  third  and  final  phase  of  the  study  area,  work  on  the  final  project  continues.  Instructor  and 
students  continue  to  communicate  via  a wide  variety  of  technologies,  including  audioconference,  email,  and  chat. 
Students  may  use  these  technologies  to  communicate  within  formal  or  informal  study  groups.  Students  use  the  same 
technologies  to  present  their  final  projects  to  the  instructor  and  classmates. 

The  Applied  Thesis  and  Dissertation 

The  1TDE  applied  thesis  and  dissertation  differ  from  their  more  traditional  cousins  in  at  least  three  ways. 
First,  they  must  offer  a solution  to  a problem  in  the  work  place.  Second,  the  processes  of  implementing  and 
reporting  are  conducting  concurrently  with  coursework,  rather  than  afterward.  Third,  students  are  guided,  not  by  a 
major  professor,  but  by  an  applied  thesis/dissertation  adviser,  who  may  be  a program  professor,  but  is  more  likely  an 
expert  identified  expressly  for  the  purpose. 

The  Technologies 

1TDE  instructors  and  students  have  used  a wide  variety  of  communication  technologies.  This  variety  has 
been  the  result  of  technological  advances,  instructor  preferences,  specific  course  needs,  and  a spirit  of 
experimentation.  To  a very  large  degree,  the  cluster  concept  has  dictated  the  choice  of  communication  technologies, 
which  must  be  easily  accessible  to  students  and  permit  student-student  interaction,  both  synchronous  and 
asynchronous.  The  principal  classes  of  communication  technologies  include  Internet  chat,  audioconference, 
videoconference,  and  Web-based  course  packages. 

Internet  Chat 

When  the  ITDE  program  was  inaugurated,  in  the  spring  of  1996,  Netscape  Chat  was  available  as  a free 
download  and  became  the  basis  for  regularly  scheduled  chat  sessions.  By  the  beginning  of  their  third  year  in  the 
program,  Netscape  Chat  had  been  replaced  by  a First  Class  Internet  client.  Embanet  is  now  used  by  four  clusters  for 
email,  chat,  and  uploading  and  downloading  of  files  (assignments,  study  guides,  required  forms)  in  study  areas, 
cluster  areas,  and  workgroups.  The  advantages  of  the  First  Class  client  include:  multiple  platform  capability;  ease  of 
use;  sophisticated  features  and,  perhaps  most  importantly;  24-hour  support,  online  and  via  telephone,  for  all 
registered  users,  including  students. 

Audiobridge 

The  use  to  the  audiobridge  continues  to  be  an  important  element  of  most  ITDE  courses.  Students  meet  with 
instructors  via  conference  call  accessed  by  a toll-free  number.  NSU  uses  a commercial  provider  to  set  up  the  calls 
and  provide  online  support  during  the  call.  Advantages  of  audiobridges  include  access  from  virtually  anywhere, 
efficiency  (faster  than  typing),  and  the  ability  to  make  tape  recordings  of  calls.  A visual  component  has  been 
addressed  by  many  ITDE  students  by  adding  a second  telephone  line  to  their  homes  so  that  they  may  access  Web- 
based  materials  during  the  audioconference. 

Videoconferencing 

Although  traditional  compressed  video  has  for  years  been  used  by  NSU  programs  (including  ITDE)  to  bring 
guest  speakers  to  students  during  face-to-face  instruction,  a trial  of  desktop  videoconferencing  has  been  conducted 
with  Cluster  One.  The  ITDE  program  provided  desktop  video  boards,  cameras,  and  software  to  all  students  and 
subsidized  the  lease  of  ISDN  telephone  lines  in  their  homes.  Desktop  video  has  been  used  as  a medium  for  the 
delivery  of  instruction,  for  discussions,  and  for  student  presentations. 

Online  Courses 

The  demand  for  online  courses  is  undeniably  substantial,  and  growing.  The  ITDE  program  has  moved 
toward  offering  online  courses,  but  slowly  and  deliberately.  Over  the  course  of  the  first  two  years  of  the  program, 
several  of  the  newer  online  course  development  packages  were  evaluated.  In  the  fall  of  1998,  WebCT  was  used  to 
deliver  an  online  course.  And  again,  the  pioneers  were  the  members  of  Cluster  One,  who  participated  in  their  final 
study  area.  Leadership  II,  online.  Four  of  the  nine  study  areas  will  be  delivered  online  in  upcoming  years. 
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The  Best  of  Both  Worlds 


After  meeting  for  several  weeks  via  videoconference,  Cluster  One  gathered  in  Fort  Lauderdale  for  an 
extended  week  of  intensive  instruction.  However,  even  during  this  face-to-face  phase  of  the  course, 
videoconferencing  technologies  were  again  used,  this  time  to  bring  guest  speakers — leaders  in  the  field  of 
instructional  technology — to  the  students.  The  presentations  were  the  catalyst  for  wide-ranging  discussions  about  the 
status  and  future  field.  In  the  weeks  that  followed,  students  and  instructor  continued  to  communicate  at  a distance. 

None  of  the  elements  of  the  Nova  ITDE  Model — the  uniform  course  of  study,  telecommunicated  distance 
education,  intensive  face-to-face  instruction,  the  cluster  concept,  or  the  applied  research  project — are,  in  themselves, 
innovative.  Rather,  it  is  the  combination  of  attributes  that  is  both  powerful  and  noteworthy. 

Graduate  programs  in  instructional  technology  and  distance  education  may  be  successfully  offered  face-to- 
face  or  at  a distance.  Tradition,  of  course,  has  favored  the  former,  while  schools — both  established  and  new — are 
increasingly  offering  courses  and  entire  programs  at  a distance.  Both  have  significant  advantages  and  disadvantages. 
However,  rather  than  choosing  between  these  two  modes  of  instruction,  the  Nova  ITDE  model  incorporates  both.  In 
so  doing,  it  achieves  the  best  of  both  worlds  through  the  determination  of  an  appropriate  ratio  of  face-to-face  and 
distance  instruction,  a balanced  set  of  technologies  used  to  deliver  instruction  at  a distance,  and  a program  structure 
that  builds  upon  the  strengths  while  overcoming  the  weaknesses  of  distance  education. 

The  choice  of  instructional  delivery  method  is  not  based  merely  upon  media  effectiveness.  Clearly, 
research  indicates  that  students,  on  average,  learn  as  well  at  a distance  as  they  do  in  traditional,  face-to-face  settings. 
Further,  decades  of  research  have  shown  that  students  learn  effectively  from  properly  designed  instruction  delivered 
via  any  medium.  The  Nova  ITDE  model  is  based  on  an  understanding  that  a “best”  balance  both  of  face-to-face  and 
distance  instruction  and  of  media  may  be  established  for  a given  group  of  students  and  instructors.  This  balance  will 
shift,  depending  on  needs  and  preferences  of  the  learner,  the  technological  expertise  of  the  instructor,  and  the 
available  technologies. 

In  part,  the  ITDE  Model  is  shaped  by  the  concept  of  equivalency,  which  holds  that  distance  education’s 
appropriate  application  should  provide  equivalent  learning  experiences  for  all  students — distant  and  local — in  order 
for  there  to  be  expectations  of  equivalent  outcomes  of  the  educational  experience.  In  advancing  equivalency,  the 
cluster  concept  provides  a practical  solution  to  some  of  the  challenges  presented  by  distance  education.  Within  the 
cluster  friendships  are  formed,  issues  are  debated,  and  support  is  offered.  In  these  and  other  ways,  the  cluster 
simulates  the  experience  of  the  graduate  assistant  “bullpen”  a not-insignificant  element  of  the  graduate  school 
experience  offered  by  traditional  programs.  Because  of  their  cluster  membership,  ITDE  students  are  able  to 
overcome  isolation,  the  truly  important  “distance”  of  distance  education.  Further,  peer  pressure  within  the  cluster 
encourages  students  to  persist,  overcoming  one  of  the  historic  drawbacks  of  distance  education. 

A final,  critical  element  of  the  ITDE  Model  is  the  applied  dissertation.  Its  emphasis  on  practical  application 
of  course  content  personalizes  learning,  and  in  so  doing,  deepens  understanding. 

The  ITDE  program  from  which  Cluster  One  will  graduate  is  strikingly  different  from  the  program  in  which 
it  began  three  years  ago.  Beginning  with  the  goal  of  achieving  the  best  of  both  worlds — traditional,  face-to-face  and 
distance  education — it  has  undergone  numerous  and  significant  modifications.  Technologies  have  been  adopted  and 
discarded,  courses  have  been  significantly  revised,  policies  have  been  adjusted,  and  the  program  has  been 
strengthened.  If  the  students  of  Cluster  One  have,  indeed,  been  “put  through  the  wringer,”  they — and  the  program — 
are. the  better  for  it.  For  both  the  institution  and  the  students,  it  has  been  a learning  process. 
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Instructional  design  is  critical  to  distance  education  and  planning  for  effective  teaching  is  needed  for 
learning  to  occur.  Greater  emphasis  needs  to  be  placed  on  the  planning  process.  The  instructional  design  process 
organizes  and  systematizes  planning.  It  is  essential  to  consider  elements  such  as  the  content,  the  learner,  the 
strategies  for  teaching,  and  the  means  for  assessing  the  learning  experience.  By  following  through  with  this  process, 
the  instructor  will  find  that  teaching  at  a distance  is  an  exciting  and  dynamic  experience.  One  that  will  be  welcomed 
by  both  the  instructor  and  the  learners. 
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Adapted  from  Dick  & Carey, The  Systematic  Design  of  InstnjctioQ  1996 

The  instructional  design  process  considers  all  aspects  of  the  learning  environment,  following  a well 
organized  procedure  that  provides  guidance  to  even  the  novice  distance  instructor.  Viewing  the  instructional 
environment  as  a system,  fosters  a relationship  among  and  between  all  the  components  of  that  system  - the  instructor, 
the  learners,  the  material,  and  the  technology.  When  this  procedure  is  followed,  attention  is  drawn  to  considering  the 
components  of  the  system.  Especially  when  planning  for  distance  education,  the  instructor  must  make  decisions  that 
will  affect  all  aspects  of  the  distant  instructional  system  (Moore  & Kearsley,  1996).  Focusing  on  the  distant  system 
will  help  the  instructor  plan  for  effective  learning  experiences  and  the  instructor  will  have  a greater  opportunity  for 
developing  a balanced  distant  learning  experience. 

Principles  of  Instructional  Design  Systems 
Systematic  Process 

With  the  goal  of  student  learning  in  mind,  an  instructor  can  consider  the  components  of  a successful 
learning  system  (Dick  & Carey,  1996).  The  interaction  of  the  components  - learner,  content,  method/material,  and 
environment  - provides  results  in  creating  the  type  of  learning  experience  necessary  for  student  learning.  These 
components  must  interact  both  efficiently  and  effectively  to  safeguard  quality  learning  experiences.  There  should  be 
a balance  among  the  components,  none  can  take  on  a higher  precedence  over  the  others.  This  equal  interaction  of 
the  system’s  components  safeguards  that  the  instruction  will  reach  the  goal  of  student  learning. 

Planning  for  Instruction  at  a Distance 

The  planning  and  organization  for  a distance  education  course  is  multifaceted  and  must  occur  well  in 
advance  of  the  scheduled  instruction.  To  eliminate  trial  and  error  preparation,  distance  learning  faculty  should 
consider  the  following: 

• Courses  previously  taught  in  traditional  classrooms  may  need  to  be  retooled.  The  focus  of  the  instruction  shifts 
to  more  visual  presentations,  engaged  learners,  and  careful  timing  of  presentations  of  information. 

• As  traditional  classroom  materials  that  have  been  used  during  instruction  are  revised,  considerations  for 
illustrating  key  concepts,  or  topics,  using  table,  figures  and  other  visual  representations  need  to  be  made. 

• Activities  that  encourage  interactivity  need  to  be  incorporated.  Planning  for  interactivity  helps  learners.  Not 
only  does  the  instructor  have  to  plan  for  interaction,  but  students  may  require  training  to  participate  actively  in 
these  types  of  distant  interactive  activities. 

• Activities  that  allow  for  student  group  work  need  to  be  well  planned.  This  helps  construct  a supportive  social 
environment.  For  example,  the  instructor  could  present  case  studies  related  to  theories  and  concepts  covered  in 
the  course,  then  groups  of  students,  perhaps  in  different  sites,  could  discuss  case  study  questions  and  reach 
consensus  on  a solution  to  the  problem. 

• Technical  problems  can  occur,  so  alternatives  for  the  class  must  be  considered.  If  equipment  fails,  it  is 
important  for  students  to  have  projects  and  assignments  independent  of  the  instructor  and  alternative  means  of 
communication  (e.g.,  fax,  phone,  e-mail).  Prior  discussion  of  the  plan  for  technology  failures  with  students  will 
eliminate  confusion  and  loss  of  productive  class  time  (Herring  and  Smaldino,  1997). 

In  addition  to  considerations  related  to  planning  for  instruction,  there  is  also  a need  to  examine  issues 
associated  with  the  separation  of  instructor  and  some  or  all  of  the  students.  Time  constraints  for  class  delivery,  lack 
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direct  contact,  visualization  of  the  materials,  and  planning  for  interaction  requires  a reconsideration  of  classroom 
dynamics.  Often  instructors  use  visual  cues,  such  as  student  facial  expressions,  within  the  traditional  classroom  or 
conversations  with  students  after  class  to  decide  quickly  to  adjust  the  instructional  approach  for  a course.  These  cues 
give  instructors  insights  that  help  them  personalize  the  instruction  for  the  students  and  insure  a quality  learning 
experience  for  all.  Teaching  at  a distance  eliminates  many  of  these  cues.  Alternative  approaches  to  on-going 
evaluation  of  instruction  must  be  incorporated.  If  instructors  ignore  this  area  of  preparation,  and  plan  to  teach  as 
they  always  have,  they  will  feel  frustrated.  Likewise,  students  may  feel  alienated  and  will  begin  to  “tune  out”  the 
instructor.  The  instructional  development  process  should  be  based  on  the  unique  characteristics  and  needs  of 
students,  meshed  with  the  teaching  style  of  the  instructor  and  the  course  goals  and  content.  Interaction  should  be 
maximized,  visual  potential  of  the  medium  should  be  explored,  and  time  constraints  addressed. 

Issues  to  Address  in  the  Planning  Process 
Who  are  the  Learners? 

Taking  the  time  to  learn  about  the  learners  in  the  class  yields  a more  productive  learning  environment. 

Along  with  the  general  information  about  the  learners,  an  instructor  needs  to  know  more  specific  information  about 
the  students  in  the  class.  Factors  such  as  whether  the  students  are  from  urban  or  rural  areas,  age  range,  grade  range, 
and  educational  background  can  have  a marked  impact  on  the  levels  of  interaction  among  students.  The  instructor 
may  have  to  plan  more  carefully  for  the  types  and  levels  of  interaction  to  ensure  a quality  learning  experience  for  all 
members  of  the  class. 

The  cultural,  social,  and  economic  backgrounds  of  the  students  is  also  important  information  for  the 
instructor  (Willis,  1994).  Educational  expectations  of  learners  can  also  influence  the  quality  of  the  learning 
experience.  The  attitude  and  interest  students  bring  to  the  class  will  impact  the  learning  environment.  Thus,  an 
instructor  who  is  interested  in  creating  a quality  learning  experience  for  all  members  of  the  class,  with  the  ultimate 
goal  of  learning  as  the  outcome,  will  be  certain  to  account  for  these  variables  in  planning. 

Students  who  are  less  social  may  find  the  distance  education  environment  more  comfortable  for  them. 
Students  may  become  more  expressive  because  of  the  perception  of  privacy  and  the  informative  nature  of  mediated 
communication.  They  may  perceive  the  increased  and  varied  interactivity  and  immediate  feedback  as  a positive 
input  to  their  interface  with  the  learning  experience. 

Additionally,  students  can  benefit  from  a wider  range  of  cognitive,  linguistic,  cultural,  and  affective  styles 
they  would  not  encounter  in  a self-contained  classroom.  The  emphasis  should  not  be  on  the  inherent  efficiency  of 
the  distance  learning,  but  on  the  values  and  services  offered  to  students  through  their  exposure  to  others  (Herring  & 
Smaldino,  1997).  Relationships  can  be  fostered,  values  can  be  expanded,  and  a shared  purposes  or  goals  can  be 
developed. 

To  be  effective,  an  understanding  of  the  target  audience  is  necessary.  Willis  (1994)  suggested  the  following 
questions  be  asked  prior  to  development  of  distance-learning  environments: 

• What  are  student  ages,  cultural  backgrounds,  interests  and  educational  levels? 

• What  is  the  level  of  familiarity  of  the  students  with  the  instructional  methods  and  technological  delivery 
systems  under  consideration? 

• How  will  the  students  apply  the  knowledge  gained  in  the  course,  and  how  is  it  sequenced  with  other  courses? 

• Can  the  class  be  categorized  into  several  broad  subgroups,  each  with  different  characteristics? 

What  is  the  Essential  Content? 

The  content  of  a course  needs  to  reflect  articulation  within  the  curriculum.  It  is  essential  to  examine  the 
nature  of  the  content,  as  well  as  the  sequence  of  information.  In  any  distance  learning  environment,  one  particular 
issue,  that  of  time  constraints,  impacts  other  planning  areas.  Time  constraints  refer  to  the  actual  on-line  time  for 
delivery,  which  is  often  limited  and  non-flexible.  The  issue  of  limited  time  makes  it  necessary  to  closely  examine  the 
essential  elements  of  the  course  content.  The  instructor  needs  to  balance  content  with  the  limited  time  for  learning 
activities  and  possibly  remove  extraneous,  nonessential  information. 

Generally  speaking,  the  scope  of  the  content  for  a course  needs  to  be  sufficient  to  ensure  the  entire  learning 
experience  will  lead  to  the  desired  outcomes.  Concepts,  knowledge,  and  specific  skills  need  to  be  identified  (Dick  & 
Carey,  1996).  Supporting  information  or  knowledge  is  important  to  the  scope  of  content  analysis.  Follow-up  and 
applications  of  the  content  should  be  considered. 

It  is  important  to  remember  that  no  matter  which  media  are  used,  the  trend  is  to  reduce  the  “amount”  of 
information  delivered  and  to  increase  the  “interactive  value”  of  the  learning  experience  (Herring  & Smaldino,  1997). 
Thus,  the  instructor  may  need  to  “throw  out”  content  that  had  been  included  in  a traditional  presentation  of  a course. 
Or,  the  instructor  may  need  to  reconsider  means  of  “delivery”  of  the  information  through  alternative  means,  such  as 
additional  reading,  booklets  designed  specifically  for  the  tasks,  links  to  special  sites,  etc. 


What  Teaching  Strategies  and  Media  to  Use? 

Successful  teaching  at  a distance  places  the  recipients'  needs  before  organizational  convenience  and  at  the 
center  of  planning  and  decision  making.  The  individual  needs  of  the  learners  are  brought  to  the  forefront  in 
education  that  uses  electronic  technology,  because  separation  of  learners  from  the  instructor  requires  students  to  take 
more  responsibility  for  learning.  Consequently,  the  learner's  opinions  and  needs  play  a more  important  role  in 
decision  making  than  is  usual  in  an  instructor-centered  environment  (Macfarlane  & Smaldino,  1997). 

It  is  oversimplified  to  suggest  that  there  is  one  better  way  to  teach  at  a distance.  In  any  given  content  area 
there  are  several  potential  ways  of  providing  a quality  learning  experience  for  the  students  (Heinich,  Molenda, 
Russell,  & Smaldino,  1999).  What  is  essential  in  deciding  which  strategy  or  strategies  to  employ  is  the  issue  of 
engaging  the  learner.  The  one  thing  that  has  been  repeatedly  demonstrated  through  research  is  that  lecture,  or  the 
“talking  head,”  approach  is  the  least  successful  strategy  to  employ  in  distance  education  (Schlosser  and  Anderson, 
1996). 

The  instructor  needs  to  focus  on  selecting  instructional  strategies  that  engage  the  learners  in  active  learning. 
To  do  this,  the  instructor  may  need  to  de-emphasize  the  “informative”  part  of  the  instruction  for  more  “discovery”  of 
information.  The  emphasis  on  keeping  the  learners  engaged  in  learning  ensures  that  students  will  be  “in  tune”  with 
the  class. 

For  the  selection  of  media  there  are  several  models  often  used  (Dick  & Carey,  i996).  One  common  theme 
with  all  of  these  models  is  the  learning  context,  which  is  the  content,  the  intended  outcome,  and  the  nature  of  the 
students.  Practical  considerations  such  as  available  resources  for  creating  media  and  the  technologies  for  delivery  of 
instruction  also  play  a hand  in  the  selection  process.  Mainly,  though,  goals  and  objectives  should  be  the  primary 
influence  on  the  selection  of  media. 

Me  Alpine  and  Weston  (1994)  have  delineated  a set  of  criteria  for  selecting  media,  whether  they  are 
commercial  media  or  media  developed  specifically  for  a particular  course.  The  first  item  on  the  list  is  to  match  the 
medium  to  the  curriculum  or  content.  Also  included  are  related  items  such  as  accuracy  of  information,  motivational 
quality,  engagement  quality,  technical  quality,  and  unbiased  nature  of  material.  These  should  be  considered  in 
selecting  media  in  order  to  match  student  needs  to  the  strategies  employed. 

Visuals  provide  a concrete  reference  point  for  students,  especially  when  they  are  engaged  in  a non-televised 
learning  experience.  Providing  visuals,  even  if  they  are  lists  of  concepts  and  ideas,  can  help  students.  Visuals  also 
help  learners  by  simplifying  information.  Diagrams  and  charts  often  can  make  it  easier  to  understand  complex  ideas. 
A visual  that  breaks  down  a complex  idea  into  its  components,  can  show  relationships  that  might  be  otherwise 
confusing  to  students.  Also,  preparing  visuals  that  serve  as  mnemonics  can  assist  student  understanding.  And, 
visuals  help  students  in  their  study.  They  can  use  them  to  prepare  for  tests  and  other  means  of  assessing  their 
learning. 

Finally,  there  are  two  very  important  additional  issues  to  be  raised.  First  is  that  of  copyright.  No  matter 
what  technologies  are  incorporated  in  the  distance  environment,  the  instructor  needs  to  respect  the  copyright 
restrictions  that  might  apply.  For  example,  in  a televised  class,  the  instructor  may  not  be  able  to  use  a video  without 
first  obtaining  permission  to  display  it  to  the  class.  In  a Web-based  class,  the  instructor  may  have  to  have  permission 
to  post  a journal  article.  An  instructor  needs  to  be  responsible  to  obtaining  copyright  permissions  where  appropriate. 

The  second  issue  is  that  of  access.  The  instructor  cannot  assume  that  all  students  at  a distance  have  equal 
access  to  resources.  Students  may  not  have  the  technologies  available.  Also,  students  may  not  have  the  facilities  at 
hand.  The  instructor  needs  to  be  certain  that  all  students  have  similar  learning  experiences,  including  the  materials. 
For  example,  if  the  instructor  wishes  students  to  use  certain  books  or  journals  for  outside  reading,  it  is  important  to 
check  with  the  local  library  to  be  sure  theses  materials  are  available. 

What  is  the  Learning  Environment? 

Educators  are  familiar  with  classroom  settings.  They  are  comfortable  with  using  the  space  available  to 
enable  learning  to  take  place.  But  it  is  when  the  classroom  shifts  into  a distance  learning  setting  that  the  environment 
becomes  a challenge  to  the  instructor.  There  are  several  important  elements  to  address  within  the  distance  learning 
environment. 

Technology, 

The  type  of  setting,  be  it  place  or  time  shifted,  will  influence  planning  decisions.  Environments  that  are 
place-shifted  are  those  that  are  synchronous  but  are  not  in  the  same  location  (e.g.  a live  video-based  distance  class). 
Those  that  are  time-shifted  are  asynchronous,  where  students  access  the  class  at  different  times.  Assessing  the  use  of 
the  technologies  in  a distant  setting  is  essential.  In  any  distance  learning  environment  the  technology  becomes  the 
element  of  most  concern  for  the  instructor. 

There  are  several  issues  associated  with  technology  when  teaching  in  a distance  learning  mode.  First  is  the 
basic  operation  of  the  equipment.  In  a televised  distance  learning  setting,  switching  between  sites  is  usually  a simple 
procedure,  but  it  does  require  time  to  acquire  the  finesse  to  operate  the  switching  buttons  smoothly.  Second,  using 
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additional  cameras  in  the  classroom  can  create  some  concern  for  the  instructor.  The  overhead  camera  needs  to  be 
focused  and  materials  lined  up  to  ensure  that  learners  in  all  sites  can  see  the  material.  Third,  the  instructor  should 
always  consider  what  the  student  should  be  viewing  during  the  lesson.  Is  it  better  to  see  the  instructor,  the  visuals,  or 
other  students?  When  a instructor  has  had  experience  with  teaching  with  the  equipment,  these  decisions  become 
automatic,  making  learning  the  foundation  for  the  decisions  made  (Herring  & Smaldino,  1997). 

In  an  Internet-based  learning  environment,  the  instructor  should  be  concerned  with  the  layout  of  the 
courseware  and  the  types  of  resources  available  to  the  students  at  the  distant  sites.  The  instructor  needs  to  be  certain 
materials  are  designed  in  a way  that  is  intuitive  for  the  various  types  of  learners.  Further  the  instructor  need  to  be 
concerned  about  student  access  to  the  appropriate  hardware  and  software  to  be  successful  in  connecting  to  the 
courseware.  And,  the  instructor  should  be  concerned  that  the  students  can  complete  the  tasks  expected  of  them. 
Finally,  the  instructor  needs  to  be  certain  they  understand  the  terminology  being  used. 

It  is  essential  the  instructor  be  prepared  with  alternatives  for  each  lesson  in  case  of  problems.  What  will  the 
students  do  during  the  lesson  time  if  the  technology  is  not  operating  properly?  The  instructor  and  students  need  to  be 
prepared  for  times  when  the  entire  technology  system  is  not  working  properly.  Pre-planned  contingencies  should 
continue  the  learning  process  even  though  the  technology  is  malfunctioning.  Alternative  lessons  must  always  be 
ready,  but  hopefully  never  needed.  And,  students  need  to  be  prepared  to  know  what  to  do  with  those  materials. 

They  must  be  designed  to  be  used  without  instructor  intervention. 

Resources 

The  second  element  to  consider  in  the  instructional  environment  is  the  resources  available  to  students. 

What  materials  will  they  have  at  hand?  What  materials  will  be  available  in  libraries  and  laboratories?  Will  students 
have  access  to  resources  for  easy  communication  with  the  instructor? 

How  to  Determine  the  Quality  of  the  Instruction? 

In  the  instructional  design  process,  formative  evaluation  becomes  an  important  aspect.  Two  questions  need 
to  be  considered.  The  first  relates  to  reflection  on  the  action  or  activity:  "Is  this  approach  going  to  work?"  (Schon, 
1987).  To  be  an  effective  educator,  it  is  important  to  consider  what  can  happen  within  an  instructional  event.  All 
experiences,  both  those  considered  to  be  positive  or  negative,  have  some  element  of  surprise.  It  may  be  expectations 
were  not  achieved;  it  may  be  a serendipitous  event  led  to  an  altogether  different,  but  pleasant,  outcome.  Whatever 
the  nature  of  the  event,  it  is  essential  to  reflect  upon  what  has  happened. 

Reflection  may  take  the  form  of  critical  assessment  of  the  events,  satisfying  curiosity  about  the  nature  of 
those  events  (Macfarlane  & Smaldino,  1997).  Reflection  may  consider  the  success  of  the  learning  situation.  It 
brings  the  instructor  into  a state  of  knowing  about  the  learning  event.  It  is  now  possible  to  move  into  the  second 
question  of  the  formative  evaluation  process,  that  of  considering  how  to  improve  the  situation. 

The  second  question  then  is,  "How  can  I make  this  better?"  The  instructor  needs  to  examine  the 
instructional  event  in  terms  of  what  worked  and  what  appears  to  have  been  a problem.  The  second  phase  of  the 
formative  evaluation  is  concerned  with  helping  the  instructor  ensure  a more  successful  educational  experience  for 
students.  The  instructor  needs  to  consider  not  only  issues  such  as  the  learning  task,  the  instructional  materials,  and 
the  teaching  strategies,  but  also  where  the  technology  may  have  played  a role  in  the  instruction. 

When  examining  effective  instruction,  it  is  important  to  look  at  the  role  technology  plays  in  instruction. 

The  instructor  should  consider  the  elements  of  technologies  and  their  effect  on  the  students.  If  a problem  occurs  with 
the  lesson  because  of  the  hardware  components  of  the  system,  what  was  the  nature  of  the  problem?  Was  the  problem 
because  of  a temporary  interference  with  the  transmission?  Was  weather  or  some  other  non-controllable  issue 
causing  problems  with  the  transmission?  Can  the  hardware  be  improved?  Can  things  be  done  to  the  interactive 
instructional  classroom  to  aid  instruction  in  the  future? 

If  a problem  does  not  relate  to  hardware,  then  what  was  the  problem?  Perhaps  students  needed  to  be  better 
informed  about  how  to  use  the  equipment.  It  may  be  that  students  needed  preparation  for  the  lesson.  Perhaps  the 
instructor  needed  to  prepare  other  types  of  handouts  or  manipulatives  to  ensure  that  the  students  could  accomplish 
the  tasks.  Maybe  the  instructor  needed  to  select  an  alternative  teaching  strategy  to  improve  interactivity  and  student 
outcomes. 

Because  so  many  different  factors  effect  the  interactive  learning  environment,  reflective  teaching  practices 
play  an  even  greater  role  in  developing  of  effective  teaching  practice.  To  consider  what  has  transpired  and  how  to 
change  it  creates  a dynamic  educational  experience  for  both  the  instructor  and  the  learners.  Formative  evaluation  is 
essential  for  successful  interactive  distance  learning  experiences. 

It  is  important  for  the  instructor  to  think  about  handouts  within  the  context  of  the  planning  process.  The 
types  of  handouts  will  vary  according  to  the  age  of  the  students  and  the  content  of  the  course.  But,  it  is  important 
that  the  instructor  realize  that  when  planning  a distant  course,  handouts  become  an  essential  communication  link  with 
students.  Therefore,  the  instructor  needs  to  invest  time  and  energy  in  creating  quality  handouts  for  students. 
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Even  within  a traditional  class,  the  instructor  is  concerned  with  getting  materials  to  the  students.  Often 
papers  and  books  are  distributed  at  the  beginning  of  the  class  period.  But,  when  teaching  at  a distance,  this  is  not  as 
easy  as  it  might  seem.  Often  the  majority  of  the  class  is  at  a distance  and  distribution  of  materials  becomes  a 
logistical  nightmare. 

An  instructor  needs  to  consider:  (a)  getting  the  materials  to  the  distant  sites  on  time.  A distribution  network 
must  be  established  for  getting  tests  and  other  materials  to  those  remote  sites.  The  technology  can  be  useful  in 
transferring  materials;  (b)  communicating  with  the  students.  The  effect  of  separating  instructor  and  students  does 
affect  this  communication;  (c)  time  delays  in  material  transfer.  Students  may  have  to  wait  a longer  time  than 
normally  expected  to  receive  written  feedback.  Instructors  may  elect  to  use  other  forms  of  telecommunications  with 
students  to  facilitate  this  feedback. 

Summary 

The  instructional  design  process  provides  the  framework  for  planning.  It  is  essential  that  the  instructor  take 
the  time  to  plan  and  organize  the  learning  experience  prior  to  implementation  when  engaged  in  teaching  at  a 
distance.  The  instruction  will  be  at  a standard  that  is  acceptable  in  all  venues.  The  students  will  be  engaged  and  the 
instructor  will  be  satisfied.  Planning  make  the  difference  in  a successful  learning  environment. 

References 

Dick,  W.  and  Carey,  L.  (1996).  The  systematic  design  of  instruction.  New  York:  Harper  Collins 
Publishers. 

Heinich,  R.,  Molenda,  M.,  Russell,  J.  and  Smaldino,  S.  (1999).  Educational  media  and  technologies  for 
learning.  Columbus,  OH:  Merrill/Prentice  Hall. 

Herring,  M.  and  Smaldino,  S.  (1997).  Planning  for  interactive  distance  education:  A handbook. 

Washington,  DC:  AECT  Publications. 

Macfarlane,  C.  & Smaldino,  S.  (1997).  The  electronic  classroom  at  a distance.  In  R.  Rittenhouse  & D. 

Spillers  (Eds.),  Modernizing  the  curriculum:  The  electronic  classroom.  Springfield,  MO:  Charles  Thomas 
Publishers. 

McAlpine  and  Weston  (1994).  The  attributes  of  instructional  materials.  Performance  Improvement 
Quarterly , 19-30. 

Moore,  M.  and  Kearsley,  G.  (1996).  Distance  education:  A systems  view.  Belmont,  CA:  Wadsworth 
Company. 

Schlosser,  C.  & Anderson,  M.  (1993).  Distance  education:  Review  of  the  literature.  Ames,  IA:  Research 
Institute  for  Studies  in  Education. 

Schon,  D.  (1987).  Educating  the  reflective  practitioner.  San  Francisco:  Josey-Bass. 

Willis,  B.  (1994).  Distance  education:  Strategies  and  tools.  Englewood  Cliffs,  NJ:  Educational 
Technology  Publications. 


209 


BEST  COPY  AVAILABLE 


219 


Using  Multimedia  to  Counter  Stereotypes  in  Science  Classrooms:  New  Perceptions  of 

Who  Becomes  a Scientist 


Abstract 


Leslie  D.  Edwards 
University  of  Colorado  — Boulder 


Despite  gains  in  many  areas,  minorities  remain  underrepresented  as  scientists  and  science  students.  One 
of  the  reasons  for  this  absence  may  be  the  presence  of  a common  stereotype  of  scientists  as  white  males.  1 
argue  in  this  paper  that  recent  research  in  cognitive  and  social  psychology  shows  that  stereotypic 
processes  are  not  automatic,  and  that  stereotypes  can  be  reversed  through  the  use  of  multimedia  software 
in  the  science  curriculum  which  includes  graphic  images  with  specific  occupation  and  background 
information  about  ethnic  groups.  Community  college  students  who  used  the  counterstereotypic  version 
showed  less  negative  stereotypic  attitudes  afterwards,  compared  with  students  using  the  same  software 
without  the  counterstereotypic  images,  when  used  within  a course  context  which  included  discussion  of 
cultural  differences.  Based  on  the  quantitative  stereotypic  attitude  changes  seen  here  I propose  a two-step 
model  for  implementing  changes  in  classroom  curriculum  which  could  influence  student  stereotypes  about 
who  becomes  a scientist  and  perhaps  lead  to  an  increase  in  the  number  of  minorities  studying  and 
practicing  science. 


Introduction 

Despite  2 decades  of  federal  programs  and  funding  to  increase  the  representation  of  women  and  minorities 
in  science,  there  have  been  few  changes  in  the  percentages  of  underrepresented  groups  in  science..  ..and  efforts  to 
raise  the  numbers  of  minority  scientists  show  no  appreciable  gains....  Eisenhart,  Finkel  and  Marion,  1996. 

“Mami,  cuando  crezca  quiero  ser  biologo  de  moleculas,  even  though  we  don’t  know  anyone  who  does  that 
job.”  The  number  of  African-Americans,  Latino/Hispanics,  and  Native  Americans  studying  and  practicing  science 
makes  it  difficult  for  children  in  some  communities  to  envision  themselves  in  certain  careers.  For  example,  in  1988 
only  3%  of  scientists  were  Latino/Hispanic  although  that  ethnic  group  comprised  7%  of  the  labor  force  (National 
Science  Foundation,  NSF,  1990)  while  African-Americans  were  even  more  underrepresented  as  3%  of  the  scientists 
and  10%  of  the  work  force.  Five  years  later,  the  percentage  of  African-American  and  Latino/Hispanic  scientists 
remained  the  same,  although  both  groups  increased  in  the  general  population1  (NSF,  1996). 

All  levels  of  school  seem  to  play  an  important  role  in  the  formation  of  who  decides  to  become  a scientist.  A 
recent  analysis  of  8th  and  10th  grade  students’  performance  in  science  using  a national,  longitudinal  database 
(NELS:88)  found  that  schools  themselves  may  be  a major  cause  for  the  absence  of  women  from  science  (Burkham, 
Lee  and  Smerdon,  1997). 


“...it  is  discouraging  for  us  to  report  that  girls  are  still  learning  less  science  than  boys.  It  is  even  more  discouraging 
to  find  that  the  gender  gap  favoring  boys  appears  to  increase  as  students  move  through  the  educational  system. 

This  suggests  that  schools  have  an  active  role  in  gender  stratification,  rather  than  simply  reflecting  societal 
influences  related  to  gender”  (p.  321). 

Schools  have  also  been  shown  to  play  an  active  role  in  the  underrepresentation  of  ethnic  groups  in  science 
(Harding,  1993;  Solorzano,  1994). 

A second  main  factor  which  contributes  to  the  underrepresentation  of  minorities  is  the  image  held  by 
students  about  who  becomes  a scientist  and  about  ethnic  groups.  Nelkin  (1987)  highlighted  how  the  mass  media 
portrays  scientists  as  overwhelmingly  white.  In  the  past  three  decades  researchers  have  linked  the  white  male 
stereotype  to  school  children  (Krause,  1977;  Chambers,  1983).  Recent  data  from  a study  of  elementary  student 
drawings  of  “a  scientist  doing  science”  confirms  the  continued  existence  of  certain  student  stereotypes  about  who 
becomes  a scientist  (Barman,  1997).  The  most  obvious  feature  of  the  1,504  drawings  was  the  prevalence  of  white 
males  as  depicted  in  69%  of  the  K-2  drawings,  80%  of  3rd-5th  grade  drawings  and  74%  of  the  6th-8th  grade 
drawings.  Scientists  in  the  1997  drawings,  however,  looked  like  “everyday”  people  instead  of  like  the  fictional 
“alien”  creatures  which  were  common  to  earlier  studies.  Barman  correlated  this  change  with  curriculum  reform  in 
science  that  emphasized  “activity-oriented  science.” 

Others  argue  that  holding  certain  stereotypes  influences  social  judgments  about  future  behavior  (Jussim, 
Fleming,  Coleman  and  Kohberger,  1996).  For  example,  Banaji,  Hardin,  and  Rothman,  (1993)  illustrated  how 
implicit  stereotyping  (“effects  attributed  to  unreportable  residues  of  prior  experience”)  occurred  by  incidental 
exposure  to  stereotypic  information  and  they  argue  that  this  type  of  stereotyping  unconsciously  influences  judgment. 
One  consequence  of  implicit  stereotyping  is  its  effect  on  minority  students.  Some  research  has  shown  that 
stereotypes  influence  academic  performance,  including  a report  that  African  Americans  receive  lower  grades  than 
whites  at  all  levels  due  to  a negative  stereotype  that  they,  are  not  as  intelligent  or  able  to  succeed  as  whites  (Wolfe 
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and  Spencer,  1996).  This  is  not  a recent  occurrence.  Three  studies  in  the  1980’s  (Lockheed,  1985)  looked  at 
science  performance  for  9-year-olds  and  13-year-olds  and  each  found  whites  outperformed  African-Americans  by  at 
least  15  percent. 

Despite  Barman’s  findings  that  elementary  and  middle  school  curriculum  changes  can  affect  the  kinds  of 
stereotypes  that  students  use  to  predict  who  will  become  scientists,  and  despite  the  research  that  stereotypes 
contribute  to  minority  underrepresentation  in  science,  few  researchers  have  looked  at  specific  curriculum  changes 
which  could  effectively  counter  existing  stereotypes  about  scientists  and  minorities  so  as  to  provide  encouragement 
for  more  minorities  to  study  and  succeed  in  science. 

In  this  paper,  I suggest  that  multimedia  software  which  includes  counterstereotypic  clip  art  can  be  used  to 
change  students’  images  of  who  becomes  a scientist.  The  first  section  reviews  recent  results  in  cognitive  and  social 
psychology  which  contradicted  the  prevailing  view  of  stereotypes  as  an  automatic,  individuated  response  and  which 
reveal  ways  that  stereotypes  can  be  reversed.  I then  illustrate  how  other  psychology  experiments  have  found  content 
about  occupation  and  socio-economic  background  to  be  effective  in  countering  racial  stereotypes.  Next  I describe 
how  those  findings  informed  the  design  of  a multimedia  software  program  about  genetics  as  an  attempt  to  counter 
existing  stereotypes  about  African-Americans.  Based  on  data  from  the  use  of  such  a program  in  a community 
college  introductory  biology  class  for  non-majors  I show  how  the  addition  of  that  software  to  the  science  curriculum 
contributed  to  changes  in  both  students’  stereotypic  attitudes  and  their  biology  knowledge,  within  certain 
pedagogical  contexts.  Finally,  I explore  how  instructional  and  curriculum  designers  can  build  upon  these  results  and 
suggest  a two-step  non-linear  model  for  implementation  of  curriculum  changes. 

Automatic  and  controlled  processes  in  stereotyping 

For  many  years,  psychologists  have  assumed  (and  persuaded  the  rest  of  us)  that  stereotyping  is  an 
“automatic”  brain  activity.  This  implied  that  both  encoding  (Banaji,  Hardin  and  Rothman,  1993)  and  retrieval  were 
unintentional,  involuntary,  effortless,  autonomous  and  occurring  outside  of  awareness  (Bargh,  1989).  To  investigate 
these  processes  psychologists  rely  on  providing  different  lengths  of  time  for  the  brain  to  respond.  Giving  an 
experimental  participant  less  than  350  ms  to  respond  assumes  an  “automatic”  cognitive  activity  while  longer  times 
are  assumed  to  indicate  more  “intentional”  processes  and  more  control. 

The  original  distinction  between  automatic  and  intentional  was  refined  to  include  activation  of  a stereotype 
(by  situational  cues  such  as  skin  color)  and  application  (where  the  perceiver  uses  the  activated  stereotypic 
information  in  a judgment).  Until  1996,  these  two  were  seen  as  sequential  and  activation  was  considered  to  be  an 
automatic  process  while  application  was  viewed  as  a controlled  (or  controllable)  process.  This  implied  that  many 
responses  which  disadvantaged  one  ethnic  group  over  another  were  automatic  and  beyond  the  individual’s  control. 

In  1996,  Irene  Blair  and  Mahzarin  Banaji  published  their  findings  about  ways  that  perceivers  might 
moderate  the  automatic  processes.  First,  they  found  evidence  that  confirmed  the  automaticity  of  some  responses  and 
then  they  investigated  the  nature  of  automaticity  itself.  They  used  a complicated  psychology  research  technique 
called  “semantic  priming”  which  is  based  on  the  belief  that  people  respond  faster  to  a target  word  if  it  is  paired  with 
a second  word  that  is  semantically  associated  with  the  first  word.  For  example,  people  would  respond  faster  to 
“nurse”  if  it  is  paired  with  “doctor”  than  if  it  is  paired  with  “tree.”  Psychology  researchers  use  this  concept  and 
measure  the  time  it  takes  people  to  react  to  the  second  word  (target  word)  of  paired  concepts.  When  two  words  are 
stereotypical ly  associated,  such  as  “dependent”  with  “Jane”,  a faster  reaction  time  indicates  a stronger  stereotype  by 
indicating  a stronger  association  between  the  two  words.3 

After  establishing  automatic  responses  to  stereotypes  using  semantic  priming  Blair  and  Banaji  changed  the 
instructions  given  to  participants  so  as  to  moderate  the  priming.  Some  subjects  were  given  a stereotypic  strategy  by 
being  told  to  expect  an  associated  second  word,  i.e.  “Jane”  would  follow  “petite”.  Others  were  instructed  with  a 
counterstereotypic  strategy  by  being  told  the  opposite,  i.e.  to  expect  that  “Jack”  would  follow  “petite.”  They  did  the 
experiments  at  both  350  ms  and  2000  ms  intervals.  The  most  startling  result  was  that  the  counterstereotypic 
instructions  reversed  the  priming  within  the  2000  ms  interval  but  not  within  350  ms. 

If  the  brain  acted  so  quickly  that  an  automatic  response  was  presumed,  it  followed  instructions  when  they 
matched  the  priming;  it  gave  faster  response  times  to  stereotypic  pairs  when  it  was  given  stereotype  instructions  and 
it  gave  faster  responses  to  counterstereotypic  pairs  when  given  counterstereotypic  instructions.  However,  when 
allowed  a longer  time  to  respond  (2000  ms),  so  as  to  include  intentional  processes  and  more  control,  the  brain  also 
followed  counterstereotypic  instructions  with  stereotypic  pairs  and  gave  slower  response  times  with  those  pairs  and 
faster  response  times  with  counterstereotypic  pairs— in  direct  contradiction  of  the  automatic  aspects  of  the 
stereotyping  process.  In  other  words,  when  the  cognitive  processes  were  more  controlled,  participants  were  able  to 
modify  the  automaticity  of  the  stereotyping  process. 

If  one  accepts  the  results  of  Blair  and  her  colleagues  that  environmental  information  can  contribute  to 
encoded  stereotypes  but  that  this  information  is  controllable  and  changeable,  the  question  then  becomes  what  kind  of 
information  could  be  used  to  counter  existing  negative  stereotypes.  Social  psychology  research  within  two  well- 

, 222 


211 


accepted  theories  (complexity-extremity  theory  and  assumed  characteristics  or  trait  attribution  theory)  sheds 
additional  insight  to  begin  to  answer  that  question.  4 

Theories  on  the  influence  of  stereotypes 

Complexity-extremity  theory  suggests  that  people  have  more  information  about  their  own  group  than  about 
other  groups  and  that  the  more  information  they  have,  the  less  extreme  their  evaluation  will  be.  Patricia  Linville  and 
Edward  Jones  (1980)  did  a series  of  experiments  which  looked  at  how  members  of  one  cultural  group  (the  “in- 
group’’) evaluated  members  of  another  cultural  group  (the  “out-group”).  In  one  experiment,  male  subjects  were 
given  a packet  of  40  cards  which  each  contained  a personality  trait  which  had  been  pre-tested  to  reflect  traits  of  both 
black  and  white  undergraduates.  They  were  told  to  sort  the  40  traits  into  groups.  Some  subjects  were  told  to  think 
about  White  male  undergraduates  while  sorting  and  other  participants  were  told  to  think  about  black  male 
undergraduates  while  sorting.  Their  all-white  participants  formed  significantly  more  groups  when  thinking  about 
whites  than  about  blacks  and  scored  higher  on  a conceptual  complexity  index  test.5 

These  results  suggested  that  whites  had  greater  schematic  complexity  regarding  in-group  members  than  out- 
group members.  The  extremity  of  an  evaluation  is  theorized  to  depend  upon  the  richness  (greater  schematic 
complexity)  of  one’s  conception  of  the  “other’s”  social  group,  that  is,  richer  information  results  in  a less  stereotypic 
attitude.  Variations  in  schematic  complexity  are  also  important  because  they  suggest  differential  influences  for  new 
information  since  a simple  and  impoverished  schematic  about  out-group  members  is  likely  to  be  heavily  impacted  by 
a small  amount  of  new  information. 

Trait-attribution  theory  (also  called  assumed-characteristics  theory  ) proposes  that  a person  with  information 
about  only  a few  traits  will  use  broad  complexes  which  include  many  traits  to  evaluate  another  person.  For  example, 
information  about  skin  color  leads  people  with  limited  information  about  the  other  individual  to  make  assumptions 
about  socio-economic  status,  personality  traits,  or  values. 

One  line  of  trait-attribution  studies  looked  at  racial  stereotypes  in  light  of  information  about  background  and 
occupation.  Jack  Feldman  (1972)  had  college  students  generate  descriptive  adjectives  for  four  categories:  “Negro”, 
“white”,  “professional”  and  “working  class”.  Other  college  students  then  selected  a percentage  for  each  adjective 
which  represented  the  number  of  people  from  each  of  the  four  categories  that  would,  in  their  opinion,  exhibit  that 
trait.  A comparison  of  white  and  black  categories  showed  that  the  occupational  stereotype  was  much  stronger  and 
more  differentiated  than  racial  stereotypes.  Professional  background  and  working-class  occupation  were  rated 
significantly  higher  on  the  black  stereotype  than  were  other  adjectives.  In  other  words,  “professional  blacks”  were 
more  professional  than  comparable  “professional  whites.” 

Feldman  argued  that  professionals  are  accorded  more  traits  and  rated  higher  on  those  traits  than  are  working 
class  people.  He  concluded  that  the  black  stereotype  is  “in  reality,  a lower-class  black  stereotype,  and  that  when 
contradictory  information  is  presented,  it  takes  precedence  over  race”  (1972,  p.  338). 

If  complexity-extremity  theory  underscores  how  small  amounts  of  new  information  about  out-group 
members  can  be  influential  and  if  trait-attribution  theory  offers  information  about  occupation  and  background  as 
effective  influences,  the  question  becomes,  what  kind  of  information  about  occupation  and  background  should  be 
provided?  Alice  Eagly  & Valerie  Steffen  (1984)  have  shown  that  the  new  information  can  be  extremely  general.  In 
their  experiment,  they  read  a short  sentence  to  participants  about  a person  (e.g.,  “Phil  Moore  is  about  35  years  old 
and  has  been  employed  for  a number  of  years  by  a supermarket.  He  is  one  of  the  managers”).  All  descriptions 
included  being  employed  by  either  a bank,  a supermarket,  a medical  clinic  or  a university  department  of  biology. 
Occupational  status  was  conveyed  by  job  title.  “Vice-president,”  “manager,”  “physician”  and  “professor”  were  used 
to  indicate  high-status  while  “teller”,  “cashier,”  “x-ray  technician”  and  “lab  technician”  indicated  low-status. 
Participants  then  rated  each  person  on  personal  attributes  indicating  communal  (selflessness  and  concern  with  others) 
and  agentic  (self-assertion  and  urge  to  master)  qualities  which  were  selected  to  reflect  gender  stereotypes.  The 
results  indicated  that  the  extremely  general  information  about  employment  status  caused  participants  to  revise  their 
estimates  of  women’s  and  men’s  communal  and  agentic  qualities  away  from  existing  gender  stereotypes. 

These  studies  suggest  that  prejudiced  responses  and  behavior  based  on  negative  stereotypes  need  not  be 
viewed  as  inevitable,  that  incidental  exposure  to  stereotypic  knowledge  can  unconsciously  and  selectively  influence 
judgment  but  that  exposure  to  counterstereotypic  information  can  moderate  the  effect  of  the  stereotype.  They  also 
imply  that  providing  information  about  a member  of  another  group  can  lessen  the  effect  of  already-held  stereotypic 
knowledge,  especially  if  the  new  information  contains  a counterstereotypic  occupation  or  background. 

Can  a similar  inference  be  made  about  the  effectiveness  of  including  certain  specific  attributes  in  a 
curriculum  as  a way  to  counter  the  effects  of  implicit  stereotypic  knowledge?  The  psychology  studies  all  involved 
experimental  laboratory  conditions  where  the  information  was  specifically  presented  to  the  brain  in  ways  that 
ensured  that  certain  cognitive  processes  would  occur  and  where  it  was  highly  likely  that  each  participant  would 
absorb  the  counterstereotypic  information.  For  educators  to  benefit  from  this  work  they  have  to  find  ways  to 
incorporate  counterstereotypic  information  into  the  curriculum  under  conditions  that  encourage  a similar  absorption. 
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To  begin  to  answer  that  question,  the  next  section  looks  at  two  studies  where  teachers  successfully  integrated 
counterstereotypic  information  into  the  classroom  curriculum. 
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Lessons  from  the  classroom 

Lorie  Strech  (1994)  looked  at  existing  racial  stereotypes  with  third-graders  in  California  before  and  after 
they  completed  a social  studies  unit  which  focused  on  the  school’s  diverse  community.  One  section  of  the 
curriculum  included  activities  to  compare  their  community  from  the  past  to  the  present.  A second  part  of  the  unit 
included  literature  which  focused  on  the  concept  of  communities  and  a third  component  was  aimed  at  dispelling 
stereotypes  by  having  students  observe  the  kinds  of  occupations  present  in  their  neighborhood  and  the  kinds  of 
people  holding  those  jobs.  Students  reported  their  data  to  the  class  as  part  of  a discussion  on  stereotypes  involving 
jobs.  An  African-American  female  psychologist  also  visited  the  class  as  a role  model. 

Students  were  given  a pre-  and  post-test  which  measured  their  association  between  certain  ethnic  groups 
(described  through  photos  of  African-Americans,  Hispanic-Americans,  Asian- Americans  and  European-Americans). 
The  curriculum  unit  was  found  to  moderate  stereotypes  with  13%  more  post-unit  responses  falling  in  “neutral” 
categories  than  in  negative  or  positive  ones.  For  example,  the  number  describing  “janitor”  as  the  occupation  for 
African-Americans  fell  from  60%  in  the  pre-unit  test  to  27%  on  the  post-unit  test  (to  be  replaced  by  “football  player” 
at  60%).  A similar  shift  appeared  for  stereotypes  of  Asian-Americans.  While  77%  of  the  pre-unit  responses  were 
“gardener”,  that  changed  to  17%  after  the  unit,  and  “teacher”  became  the  most  prevalent  with  57%.  6 

In  another  study  Richard  Johnson  and  Joseph  Tobin  (1992)  used  a videodisk  module  in  a University  of 
Hawaii  education  class  to  help  pre-service  teachers  understand  stereotypes  underlying  classroom  discipline.  The 
software  included  interviews  with  a Samoan  social  worker,  a Chinese  high  school  student  and  a Japanese- American 
professor  which  illustrated  a variety  of  perspectives  about  how  different  cultural  groups  are  stereotypic  ally  treated  in 
the  classroom.  The  authors’  conclusions,  based  on  qualitative  analysis  of  student  responses  and  critiques,  suggest 
that  the  education  students  changed  some  of  their  stereotypes  after  viewing  the  program.  The  change  in  stereotyping 
information  came  about  when  pre -service  teachers  were  provided  rich  information  about  three  varied  ethnic  groups. 

While  these  two  projects  support  the  effectiveness  of  a small  curriculum  modification  for  influencing  the 
use  of  stereotypic  information,  they  concerned  social  studies  and  education  courses  where  the  content  could  naturally 
include  a focus  on  culture.  In  science,  the  adoption  of  strategies  designed  to  narrow  the  gender  gap  in  science  (i.e., 
increasing  student  confidence,  student  performance  and  student  interest  in  science,  as  suggested  by  Burkham,  Lee 
and  Smerdon,  1997)  has  failed  to  change  the  fundamental  perception  by  students  about  who  becomes  a scientist,  as 
seen  in  the  1997  study  of  elementary  school  drawings.  While  the  importance  of  these  programs  should  not  be 
discounted,  it  appears  that  we  also  need  to  find  another  way  to  introduce  counterstereotypic  information  into  science 
(and  other)  classrooms. 

One  of  the  obvious  possibilities  is  to  use  pictures  since  they  have  previously  been  shown  to  enhance 
learning  in  certain  contexts  and  they  can  quickly  convey  large  amounts  of  detailed  information.7  Here,  in  contrast  to 
the  issue  usually  facing  instructional  designers  of  which  picture  to  use,  we  know  the  information  that  should  be 
included,  and  can  rely  upon  the  established  principle  that  pictures,  when  used  appropriately,  effectively  enhance 
learning.  The  Johnson  and  Tobin  videodisk  relied  heavily  on  images  through  the  use  of  video  for  capturing 
interviews  with  three  people  so  their  comments  could  be  available  for  student  reflection  outside  classtime.  Although 
the  presence  of  role  models  and  the  in-class  appearance  of  experts  might  be  preferable,  the  Johnson  and  Tobin  study 
shows  certain  advantages  of  technological  presentation  of  images  and  counterstereotypic  information  for  science 
classrooms.  Digital  resources  are  not  dependent  upon  a teacher’s  personal  knowledge,  cultural  experiences  or 
contacts  in  diverse  communities.  Science  is  also  especially  receptive  since  it  is  one  of  the  subjects  where  success 
requires  familiarity  and  mastery  of  technology  (Songer,  1989). 

The  technology  used  by  Johnson  on  the  videodisk  included  a wide  variety  of  elements  which  would 
generally  qualify  as  “multimedia”,  a term  for  software  programs  that  combine  text,  graphics,  animation,  audio,  video 
and  interactivity  (Lopuck,  1996).  Classroom  multimedia  can  provide  students  with  opportunities  for  independent 
learning,  problem-solving,  communication  and  self-assessment  (Platt,  Obenshain,  and  Friedman,  1994).  The  use  of  a 
multimedia  program  which  included  counterstereotypic  images  would  allow  a student  to  absorb  counterstereotypic 
information  at  her  or  his  own  pace  and  comfort  level. 

Successful  use  of  multimedia  in  the  curriculum  has  been  shown  for  anchored  instruction  (Bransford,  et.  al., 
1990),  as  part  of  cognitive  flexibility  (Spiro,  et.  al.,  1992)  and  within  a cultural  constructive  perspective  (Peters  and 
O’Brien,  1996).  The  multisensory  elements  have  been  shown  to  address  a diversity  of  learning  styles  (Dunn,  1990) 
and  to  attract  students’  attention  and  keep  them  engaged.  Static  computer  graphics  have  been  shown  to  facilitate 
learning  (Alesandrini,  1987)  and  research  on  the  movement  of  graphics  (animation)  has  also  shown  direct 
connections  to  enhancing  learning  (Enyedy,  1997;  Windschitl  and  Andres,  1998). 

Perhaps  more  importantly  for  the  purpose  of  conveying  information,  multimedia  typically  includes  a lot  of 
graphics.  Graphics  are  used  to  illustrate  ideas  in  multimedia,  as  part  of  animations,  as  interactivity  button  icons  and 
to  communicate  navigation  (e.g.,  left-pointing  arrow  means  “go  back”).  This  expectation  provides  a myriad  of 
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unobtrusive  opportunities  for  including  counterstereotypic  images.  In  short,  multimedia  appears  to  provide  a 
potential  curriculum  resource  which  can  easily  include  counterstereotypic  images  which  convey  occupation  and 
background  information.  Could  the  use  of  such  software  change  attitudes  of  students  toward  members  of  an  out- 
group? 

Methodology 

Software 

To  investigate  this  question  three  versions  were  created  of  a multimedia  software  program.  All  versions 
were  designed  to  be  used  as  a review  for  a unit  test  which  included  Mendelian  inheritance  genetics  by  helping 
students  check  their  own  comprehension  in  a game  format.  Each  had  the  same  content,  divided  into  five  sections  and 
accessed  from  a main  menu.  The  content  was  designed  to  be  easier  at  the  beginning,  with  the  first  section  containing 
vocabulary  words.  The  next  four  sections  were  alphabetically  labeled  with  questions  on  monohybrid  crosses, 
dihybrid  crosses,  x-linked  genes  and  multiple  alleles.  There  were  five  questions  for  each  section  plus  a bonus.  All 
versions  included  a total  of  30  questions. 

The  software  was  made  with  a multimedia  authoring  program  (ToolBook  by  Asymetrix,  Inc.)  which  uses 
the  metaphor  of  a book-  The  information  on  each  screen  appears  as  a “page”  which  remains  until  the  user  navigates 
(usually  by  clicking  a button)  to  the  next  “page”  or  screen.  The  software  is  also  object-oriented,  so  information  is 
presented  using  static  objects  such  as  text  fields,  created  within  the  program,  or  by  importing  graphic,  audio  or  video 
files.  All  objects  can  be  sized  or  moved  to  create  animation.  The  software  is  also  event-driven,  so  that  user  actions 
such  as  moving  the  mouse  on  top  of  a screen  object,  clicking  an  object  or  entering  a “page”  causes  other  actions  to 
occur,  such  as  a text  field  appearing  (i.e.,  to  provide  feedback)  or  a circle  moving  (i.e.,  to  look  like  a bouncing  ball). 

The  versions  differed  in  the  basic  way  they  presented  the  information.  The  first  version  (hereafter  called  the 
Web-page  based  version)  included  only  multiple  choice  questions  created  by  the  textbook  publisher.  The  first  18 
questions  were  copied,  word-for-word,  from  the  publisher’s  Web  page.  Once  a student  selected  an  answer,  one  of 
two  text  fields  appeared:  “Your  answer  to  question  #1  was  correct”  or  “Your  answer  to  question  #1  was  incorrect. 
Please  refer  to  page  239  in  [name  of  textbook  author].”  The  last  12  questions  were  taken,  also  word-for-word,  from 
the  back  of  the  textbook  chapter  and  showed  the  question  along  with  a button  which  accessed  the  answer. 

The  second  version  of  the  software  was  a “game”  (hereafter  called  plain  multimedia  version)  where  students 
scored  points  by  answering  questions.  A wrong  answer  triggered  an  audio  file  of  a dog  howling  along  with  the 
appearance  of  a text  field  as  immediate  feedback  which  included  the  correct  answer  and  a short  explanation.  A 
correct  answer  brought  applause  and  access  to  a “game”  which  included  either  archery,  bowling,  pool,  basketball  or 
darts,  depending  on  the  content  section.  Points  accumulated  and  a running  total  was  shown  on  subsequent  pages. 

The  third  version  (hereafter  called  the  counterstereotypic  version)  included  ethnocentric  clip  art  and  had 
two  differences  from  the  plain  multimedia  version.  It  included  five  images  taken  from  a commercial  “Afrocentric” 
clip  art  collection  where  the  characters  were  all  African-Americans.  Images  were  selected  which  showed  high-status 
occupation  and  background  information  or  could  be  modified  to  include  such  information.  Afrocentric  clip  art  was 
chosen  to  correspond  with  the  stereotype  measure  questions  which  asked  participants  about  “Blacks.”  For  example, 
identification  as  a doctor  or  holding  a Ph.D.  was  designed  to  convey  a high  status  occupation  or  background.8  Each 
image  was  positioned  around  the  question,  usually  on  the  left,  so  as  to  be  seen  when  the  student  read  the  question.  A 
second  difference  was  that  a popular  Mexican  game  similar  to  Anglo-Saxon  Tarot  cards  was  used  instead  of  archery 
in  one  section  and  Wari,  a common  African  game  was  substituted  for  bowling  in  another  section. 

Classroom  Setting 

The  study  was  conducted  with  two  sections  of  a 4-credit  course  for  non-majors  at  an  urban  community 
college  in  a major  city  of  the  Southwest.  Students  take  this  course  to  fulfill  their  general  science  requirement  for  an 
Associate  of  Arts  or  Associate  of  General  Studies  degree.  Most  had  graduated  from  high  school  but  some  earned 
G.E.D.s;  the  youngest  was  19  years  old  and  the  oldest  was  in  her  40’s.  The  campus  is  notable  for  the  diversity  of 
student  ethnic  backgrounds  and  in  this  study,  there  were  14  whites  (self-identified),  13  Hispanics,  2 Native 
Americans,  7 African-Americans,  2 Africans,  one  Thai  and  4 Asians.  One  class  met  in  the  evenings  and  included 
older,  “second-career”  students  while  the  other  section  was  an  afternoon  class  with  more  younger  students. 

Forty  students  in  two  class  sections,  including  one  that  I taught,  used  the  software  individually  in  a 
computer  lab  as  part  of  a regular  class  laboratory  exercise.  While  each  section  dealt  with  genetics  at  different  times 
in  the  semester,  both  spent  the  same  number  of  class  periods  and  both  used  the  software  as  a review  during  the  week 
before  a unit  test.  Prior  to  entering  the  computer  lab,  students  were  orally  informed  of  their  rights  concerning 
consent  and  participation  and  completed  two  surveys,  one  on  computer  attitudes  (Kay,  1989)  and  a second  one  on 
stereotypes.  The  stereotype  scale  had  been  created  with  10  pro-Black  and  10  anti-Black  statements  and  previously 
validated  with  174  white  students  at  Brooklyn  College  and  Lehman  College  (New  York  City)  using  principal- 
components  factor  analysis  with  varimax  rotation  (Katz  and  Hass,  1988).  An  example  of  a pro-Black  statement  is 
“Black  people  do  not  have  the  same  employment  opportunities  that  Whites  do”  and  an  example  of  an  anti-Black  item 
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would  be  “On  the  whole,  Black  people  don't  stress  education  and  training.”  Four  items  (two  of  each  type)  were 
keyed  in  reverse.9 

After  completing  the  questionnaires,  the  students  moved  to  a computer  lab,  started  the  program,  entered  a 
code  number,10  selected  their  gender  and  answered  the  question  “Briefly  describe  your  cultural  background  by  typing 
in  the  gray  box”.  The  next  five  screens  displayed  five  genetics  problems,  with  multiple  choice  answers,  which 
functioned  as  the  biology  knowledge  pre-test.  Leaving  the  last  question  triggered  a random  assignment  to  one  of  the 
three  versions,  based  on  the  time  in  seconds.  Upon  completion  of  the  review  portion,  each  student  took  a post-test 
with  five  different  genetics  problems,  exited  the  software  and  again  completed  the  attitude  toward  computers  and 
stereotypic  attitude  questionnaires  (in  the  computer  lab).  The  software  also  created  a “log”  for  each  student  which 
recorded  the  time  and  content  of  events  such  as  entering  a page  or  typing  an  answer. 11 

Results 

The  pre-  and  post-software  attitude  measures  were  converted  from  Likert  scale  to  numeric  values  such  that 
higher  scores  indicated  more  positive  attitudes  toward  a member  of  an  out-group  or  toward  computers.  Initially,  a 
3x2x4X2  factorial  design  was  used  to  analyze  the  data  using  SPSS  8.0  for  Windows  with  a four-factor  analysis  of 
variance  (ANOVA)  with  multiple  variables  including  all  three  dependent  variables  (stereotypic  attitude,  computer 
attitude  and  biology  knowledge)  and  with  software  version  (3  levels),  gender  (2  levels),  ethnicity  (4  levels:  white, 
Hispanic,  black,  Asian),  and  class  section  (2  levels)  as  independent  variables.  However,  since  there  were  no 
significant  results  for  ethnicity  as  a factor  (or  for  interactions  involving  ethnicity)  a second  ANOVA  was  used  with 
the  same  three  dependent  variables  but  with  only  software  version  (TX)  and  gender  as  fixed  factors.  The  two-factor 
ANOVA  is  shown  in  Table  1.  It  provides  a more  accurate  description  of  this  study  because  the  sample  size  is  small 
compared  with  the  number  of  cells  and  the  two-factor  ANOVA  eliminated  the  zero  frequency  cells  without  losing 
any  significant  information. 
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Table  1:  Analysis  if  Variance  with  Only  Software  and  Gender  as  Independent  Variables 


Source 

Dependent  Variable 

Type  111 
Sum  of 
Squares 

df 

Mean  Square 

F 

Sig. 

Corrected  Model 

CAMDIFF 

.636“ 

7 

9.084E-02 

315 

.97 

STERDIFF 

3.012b 

7 

430 

3.273 

.01 

QDIFF 

25.985c 

7 

3.712 

2.727 

.03 

Intercept 

CAMDIFF 

1.1 03E* 

1 

1.103E-03 

.003 

.96 

03 

STERDIFF 

.789 

1 

.789 

5.998 

.02 

QDIFF 

1.293 

1 

1.293 

.950 

.34 

TX 

CAMDIFF 

7345E- 

2 

3.673E-02 

.087 

.91 

02 

STERDIFF 

1.013 

2 

307 

3354 

.03 

QDIFF 

2.588 

2 

1.294 

.950 

.40 

GENDER 

CAMDIFF 

4.954E- 

1 

4.954E-03 

.012 

.91 

03 

STERDIFF 

1.880 

1 

1.880 

14300 

.00 

QDIFF 

1.024 

1 

1.024 

.752 

39 

CLASS 

CAMDIFF 

1.132E- 

1 

1.132E-02 

.027 

.87 

02 

STERDIFF 

311 

1 

311 

2363 

.14 

QDIFF 

2.263 

1 

2.263 

1.662 

.21 

TX*  GENDER 

CAMDIFF 

.266 

1 

.266 

.629 

.43 

STERDIFF 

.427 

1 

.427 

3.250 

.08 

QDIFF 

1.347 

1 

1347 

.990 

33 

TX*  CLASS 

CAMDIFF 

8.058E- 

1 

8.058E-03 

.019 

.89 

03 

STERDIFF 

2.214E- 

1 

2.214E-02 

.168 

.68 

02 

QDIFF 

13333 

1 

13333 

9.794 

.00 

GENDER* CLASS 

CAMDIFF 

4.011E- 

1 

4.011E-04 

.001 

.97 

04 

STERDIFF 

1.089E- 

1 

1.089E-02 

.083 

.77 

02 

QDIFF 

2.727 

1 

2.727 

2.003 

.17 

TX*GENDER*CLASS  CAMDIFF 

.000 

0 

STERDIFF 

..000 

0 

QDIFF 

.000 

0 

Error 

CAMDIFF 

8.036 

19 

.423 

STERDIFF 

2.498 

19 

.131 

QDIFF 

25.867 

19 

1361 

Total 

CAMDIFF 

8.674 

27 

STERDIFF 

5.511 

27 

QDIFF 

52.000 

27 

Corrected  Total 

CAMDIFF 

8.672 

26 

STERDIFF 

5.510 

26 

QDIFF 

51.852 

26 

a.  R Squared  a .073  (Adjusted  R Squared  = .268) 

b.  R Squared  = .547  (Adjusted  R Squared  = 380) 


The  two-factor  ANOVA  showed  significant  main  effects  for  both  software  version  and  gender  at  the  95% 
confidence  level  for  scores  on  the  stereotyping  attitudes  scale  (F=3.854,  p=.039  and  F=14.3,  p=.001,  respectively) 
but  not  for  either  attitudes  toward  computers  (CAMDIFF,  F=.087,  p=.917)  or  for  biology  question  scores  (QDIFF, 
F=.950,  p=.404).  However,  there  was  also  a statistically  significant  interaction  between  class  section  and  software 
version  (F=  9.794,  p=.006)  on  the  biology  scores. 

The  significant  software  version  differences  correlated  use  of  the  counterstereotypic  version  with  positive 
changes  in  stereotypes.  Overall  mean  scores  for  those  who  used  the  Web-page  based  software  version  dropped  - 
.3833  between  the  pre-  and  post-  results  while  those  using  the  plain  multimedia  had  mean  scores  that  stayed  about 
the  same  (a  raw  score  change  of  +.003).  Those  using  the  software  with  counterstereotypic  clip  art  saw  a positive 
change  in  their  stereotype  attitudes  of  +.1656  on  average.  These  mean  scores,  however,  differed  by  gender  with 
women  generally  scoring  more  positively  on  all  versions  and  men  scoring  more  negatively. 

The  significant  software  version  by  class  interaction  on  the  biology  questions  showed  different  trends 
between  the  two  classes,  some  of  which  may  be  accounted  for  by  the  small  sample  size.  For  example,  no  one  in  my 
class  was  randomly  assigned  to  the  Web-page  based  version.  For  those  in  my  class  who  used  the  plain  multimedia 
version,  their  biology  scores  dropped  an  average  of  -.7778  and  those  using  the  counterstereotypic  version  increased 
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+1.5  on  average  (a  30%  increase).  For  the  other  class,  the  Web-page  based  users  saw  no  change  in  scores,  those 
using  plain  multimedia  saw  an  increase  in  post-software  scores  of  +.3  and  those  using  the  counterstereotypic  version 
saw  a decrease  in  biology  scores  of -1.0  (a  20%  drop).  The  graphs  in  Figure  7 illustrate  the  class  by  software  version 
differences.  ANOVA  post  hoc  LSD  tests  showed  that  the  only  statistically  significant  difference  was  between  the 
Web-page  based  and  the  counterstereotypic  versions  (p=.042).  Table  3 shows  these  results. 

Both  the  significance  and  the  pattern  were  unexpected  results  since  there  were  only  five  questions  and  since 
it  was  expected  that  software  designed  for  review  purposes  would  be  used  by  students  who  had  already  mastered 
most  of  the  material.  For  example,  several  students  commented  at  the  end  of  the  class  period  that  the  software 
helped  point  out  the  material  they  did  not  yet  know.  I had  not  expected  the  design  to  be  sensitive  enough  to  detect 
changes  in  biology  knowledge. 

The  differential  biology  content  scores  make  sense  in  light  of  the  significant  class  by  software  version 
interaction  and  the  overall  increase  in  stereotypic  attitude  scores.  Although  the  differences  between  the  two  classes 
on  stereotype  attitudes  were  not  statistically  significant  (F=2.363,  p=.  141)  the  patterns  were  very  different.  The 
average  for  all  software  versions  in  my  class  showed  no  improvement  in  stereotypic  attitudes.  In  the  other  class 
section,  however,  group  means  for  those  using  the  Web-page  based  and  the  plain  multimedia  versions  showed 
essentially  no  change  in  pre-  to  post-software  scores  (+.075  and  +.065,  respectively)  but  the  group  using  the 
counterstereotypic  version  showed  a +.3567  improvement.  In  short,  the  use  of  counterstereotypic  clip  art  in  my  class 
correlated  with  an  increase  in  biology  knowledge  and  the  use  in  the  other  class  correlated  with  a more  positive 
stereotypic  attitudinal  change.  What  does  that  indicate  about  using  counterstereotypic  clip  art  in  the  classroom? 

These  classes  were  selected  because  they  were  the  only  sections  offered  of  this  course.  Although  the 
research  design  did  not  anticipate  differences  between  classes,  these  results  make  sense  in  light  of  the  different  ways 
culture  was  included  in  the  two  classes.  The  other  teacher  said  she  did  not  discuss  culture  during  classtime  because 
she  did  not  know  how  to  do  that.  She  encouraged  students  to  get  to  know  their  classmates,  but  did  not  conduct  any 
class  activities  designed  for  that  purpose.  For  example,  she  did  not  have  students  introduce  themselves  to  each  other 
on  the  first  day  of  class.  She  said  she  did  not  think  it  was  feasible  considering  the  number  of  students  in  the  class  and 
the  room  arrangement.  Instead  her  first  day  strategy  was  to  reassure  students  that  she  was  sympathetic  to  them  and 
knew  what  they  were  going  through  since  she  was  a student  herself.  She  also  told  them  about  things  she  had  done 
with  students  during  previous  semesters  which  showed  how  she  had  gone  out  of  her  way  to  help  students  succeed. 

In  contrast,  I encouraged  student  interactions  around  culture  in  my  class  throughout  the  semester.  For 
example,  on  the  first  day  of  class,  I asked  students  to  interview  each  other,  in  pairs,  and  to  then  introduce  their 
partner  to  the  class.  Before  we  broke  into  pairs,  I had  the  class  brainstorm  a list  of  questions  they  could  ask  of  each 
other  as  “professional  students”.  I added  the  last  question  to  the  list,  “What  is  your  cultural  background?”  After  all 
the  students  had  introduced  their  partner,  I answered  the  same  questions  for  them,  telling  them  that  my  great-great- 
great-great-great  grandmother  Butler  was  a member  of  the  Cherokee  nation. 

I also  included  a number  of  overt  discussions  throughout  the  semester  which  focused  on  student  cultural 
backgrounds  and  which  tried  to  value  differences.  On  one  occasion,  I asked  the  two  students  from  Africa  to  tell  the 
class  about  each  of  their  countries.  The  student  from  Thailand  told  the  class  about  her  country  on  another  day  and  I 
then  related  a five-minute  version  of  how  the  Hmong  came  from  China  via  Laos  and  Thailand  to  the  U.S.,  and  that 
there  were  4,000  Hmong  currently  living  in  Colorado. 

Other  conversations  were  more  directly  connected  to  biology.  For  example,  we  talked  in  class  about  sickle 
cell  anemia  which  disproportionately  affects  Africans  and  African-Americans,  about  diabetes  and  its  influence  on 
Native  Americans,  and  about  the  impact  of  Tay-Sachs  on  Ashkenazi  Jews.  One  of  the  African  students  told  the  class 
about  malaria  in  his  country.  Interviews  and  observation  data  from  the  other  class  section  indicates  that  they 
discussed  sickle-cell  anemia  as  disproportionately  affecting  African-Americans  and  Tay-Sachs  as  prevalent  among 
certain  Jewish  groups  but  failed  to  relate  diabetes  to  Native  Americans. 

The  differences  in  class  context  make  it  likely  that  if  a student  had  not  previously  been  exposed  to 
discussions  which  raised  the  issue  of  cultural  differences  in  either  an  affirmative  or  value-neutral  manner,  seeing 
counterstereotypic  images  for  the  first  time  could  be  expected  to  make  an  immediate  impression  which  could 
reasonably  appear  as  more  positive  scores  on  the  post-software  stereotype  attitude  scale,  which  is  what  was  seen  for 
those  in  the  other  class  section.  The  students  in  that  class  who  used  the  counterstereotypic  version  showed  the 
greatest  increase  in  more  favorable  stereotypic  attitudes  (group  mean  rose  from  3.94  to  4.3).  For  students  in  my 
class,  who  had  previously  discussed  cultural  differences  in  a school  setting,  seeing  cosmetic  counterstereotypic 
images  caused  no  dent  in  their  stereotypes,  but  did  contribute  to  increased  success  in  biology  scores. 

It  is  also  interesting  that  ethnicity  did  not  have  either  a significant  main  effect  or  interaction  effect, 
indicating  that  it  operated  as  an  independent  factor.  In  other  words,  the  influence  of  the  counterstereotypic  images  in 
the  software  was  similar  for  all  ethnic  groups. 
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Discussion 


The  class  by  software  version  interaction  and  its  possible  roots  in  the  context  of  the  course  are  both 
consistent  with  theoretical  discussions  in  educational  research  of  situated  cognition  (e.g.,  Brown,  Collins,  & Duguid, 
1989)  and  situated  learning  (e.g.,  Lave  & Wenger,  1991).  These  approaches  direct  attention  to  the  apparent  success 
of  learning  that  occurs  in  group  activities  which  extend  beyond  individual  cognitive  moments  and  reinforces  the 
necessity  of  viewing  learning  within  the  social  context  in  which  it  occurs. 

These  results  are  also  consistent  with  research  in  social  psychology  around  “outcome  dependency”  which 
exists  when  one  person  depends  upon  another  to  reach  a goal  or  desired  outcome.  These  psychologists  have  studied 
stereotypes  in  the  social  context  of  two  people  working  toward  a common  goal  and  have  shown  that  category-based 
stereotypes  can  be  undermined  by  individuating  processes  of  impression  formation  which  occur  when  two  people 
work  toward  a common  goal.  In  other  words,  people  working  together  will  use  specific  information,  rather  than 
general  information  derived  from  racial  categorization,  to  form  an  impression  of  the  person  with  whom  she  or  he  is 
working. 12 

The  specific  information  included  in  the  software  used  here  was  cosmetic,  with  images  which  were  neither 
integrated  with  the  genetics  content  nor  part  of  the  overt  affective  learning  goals  of  either  class  section.  Yet,  within  a 
context  of  class  discussion  of  cultural  differences,  such  minor  changes  appeared  to  immediately  influence  the 
attitudes  of  students.  The  idea  that  class  section  differences  may  stem  from  differences  in  classroom  interactions 
among  students  is  consistent  with  trait  attribution  theory.  The  less  that  a student  knows  about  another  student  who  is 
an  out-group  member,  the  more  likely  they  are  to  automatically  make  assumptions  and  rely  on  stereotypic 
information.  Introducing  students  to  each  other,  encouraging  them  to  work  together,  and  using  counterstereotypic 
clip  art  appear  to  moderate  this  tendency. 

These  results  have  several  implications  for  instructional  and  curriculum  designers.  First,  science  curricula 
could  begin  to  include  images  of  people  of  color  which  portray  them  in  high-status  occupations  and  with  upper 
socio-economic  backgrounds  as  a way  of  providing  students  with  information  which  counters  existing  societal 
implicit  stereotypes.  Second,  inclusion  of  cosmetic  images  is  not  alone  sufficient  to  create  changes  in  student 
stereotypes.  Other  educational  reforms,  such  as  class  discussions  on  culture  and  how  it  relates  to  science  need  to  be 
coupled  with  the  use  of  such  graphics. 

Third,  while  overall  results  did  not  include  a statistically  significant  effect  of  the  counterstereotypic 
software  version  on  biological  knowledge,  the  significant  class  interaction  and  the  overall  changes  in  group  means 
suggest  a tentative  model  for  changing  student  perceptions  of  who  becomes  a scientist.  The  initial  exposure  of 
students  to  counterstereotypic  information  impacts  their  implicit  stereotypes.  This  step  was  seen  with  the  other  class 
section  where  exposure  to  the  graphics  correlated  with  more  positive  student  stereotypes.  In  the  second  step, 
additional  counterstereotypic  information  absorbed  by  a student  who  had  already  experienced  some 
counterstereotypic  information,  such  as  during  class  discussions,  helps  students  to  be  more  successful  in  mastering 
science  content.  This  step  was  seen  with  my  class  section  where  a second  exposure  to  counterstereotypic 
information  correlated  with  an  increase  in  biology  scores.  This  model  is  not  linear,  however,  as  the  effect  of 
encoding  incidental  exposure  to  stereotypic  information  present  in  society  probably  needs  to  be  continually 
moderated  through  exposure  to  counterstereotypic  information.  This  further  supports  the  necessity  of  incorporating 
counterstereotypic  information  across  all  disciplines  so  that  changes  in  stereotypes  and  success  in  content  matter  are 
given  a chance  to  increase  the  attractiveness  of  studying  science  for  minorities,  a beginning  toward  increasing 
minority  representation  in  the  practice  of  science. 

Notes 

Acknowledgments.  I would  like  to  thank  Maria  Franquiz,  Lerita  Coleman,  Bob  Linn  and  Irene  Blair  for  their  very 
helpful  comments  on  an  earlier  version  of  this  paper. 

1.  While  3.5%  of  scientists  were  African-American  that  group  comprised  12%  of  the  population.  Latino/Hispanics  lost  ground;  they 
were  2.8%  of  the  science  and  engineering  workforce  while  growing  to  10%  of  the  total  population  (Committee  on  Equal  Opportunity 
in  Science  and  Engineering,  CEOSE,  1996).  That  study  also  found  that  Blacks  and  Hispanics  each  accounted  for  only  2 percent  of 
those  working  in  science  with  Ph.D.’s  while  Native  Americans  accounted  for  less  than  one  half  of  1 percent. 

2.  Psychologists  who  investigate  ethnic  stereotyping  believe  that  experiments  which  use  gender  as  the  basis  for  stereotyping  are  more 
reliable  since  they  omit  the  possibility  that  subjects  are  reacting  emotionally  to  race  issues. 


3.  There  is  another  significant  aspect  to  Blair’s  research.  The  short  time  period  she  used  (350  ms)  is  considered  by  psychologists  to  be 
a condition  of  “high  cognitive  constraint”,  meaning  that  the  person  has  very  few  mental  resources  available  for  the  task  at  hand.  Kim 
& Baron  (1988)  suggest  that  conditions  which  tax  cognitive  capacity  heighten  stereotypic  processing.  Overload,  fatigue,  low  task 
confidence,  intoxication,  drugs,  threat,  heat,  incentives,  sensory  distraction,  lack  of  structure,  temporal  constraints,  and 
embarrassment  have  been  shown  to  exacerbate  stereotypic  bias. 
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4.  There  is  a third  main  theory  about  how  stereotypes  influence  evaluative  judgment,  called  expectancy-violation  theory.  This 
perspective  suggests  that  when  one’s  characteristics  violate  stereotype-based  expectations,  the  evaluations  are  extreme  in  the  direction 
of  the  violation  (away  from  the  stereotype).  For  example,  if  a Black  has  a quality  that  is  more  positive  than  expected,  she  or  he  will 
be  rated  more  favorably  that  others  with  similar  characteristics.  Although  the  work  which  led  to  this  theory  is  consistent  with  the 
ideas  proposed  here,  expectancy-violation  was  rejected  as  too  general  and  offering  too  many  possibilities  to  be  useful  in  designing 
software. 

5.  Coleman,  Jussim  and  Kelley  (1995)  established  similar  responses  for  blacks. 

6.  The  primary  Hispanic-American  shift  was  toward  “happy”  and  “nice”  for  Hispanic  males.  On  the  pre-test,  17%  marked  “happy” 
and  20%  marked  “nice”  while  83%  marked  “happy”  and  73%  choose  “nice”  on  the  post-test. 

7.  Some  of  the  clip  art  had  to  be  edited  (i.e.,  a stethoscope  was  drawn  around  the  neck  of  a woman  since  the  Afrocentric  collection  did 
not  include  the  image  of  a woman  doctor.  These  are  not  new  ideas.  Guidelines  from  a major  publisher  in  the  1970s  (“Guidelines  for 
Creating  Positive  Sexual  and  Racial  Images  in  Educational  Materials”,  suggested  ways  to  “avoid  racial  and  minority  stereotypes  in 
art”  and  includes  ways  to  portray  African  Americans:  avoid  “skinny  geniuses  wearing  glasses”,  show  some  African  shirts,  show 
luxury  apartments  for  urban  blacks,  feature  blacks  as  the  focus  of  a picture  and  represent  blacks  in  all  professions,  including 
medicine,  law,  and  business.  See  also  Henriksen  and  Patton,  1976. 

8.  Paivio  (1971,  1978)  argued  that  the  use  of  two  independent,  interacting  cognitive  systems,  a verbal  one  for  processing  text  and  an 
imaging  system  for  processing  pictures,  explains  the  increased  retention  of  information  in  pictures.  Building  on  Paivio’s  “dual 
coding  theory”,  Kulhavy,  Stock  & Kealy  (1993)  developed  the  idea  of  verbal  and  imagery  systems  which  were  interrelated  by 
associative  links. 

9.  The  original  research  on  this  measurement  instrument  used  only  African-American  stereotypes  and  the  validation  study  included 
only  data  from  white  participants  on  the  assumption  that  attitudes  toward  a member  of  one’s  own  group  are  significantly  different 
from  attitudes  about  members  of  an  “other”  group.  Here,  after  reviewing  the  questions,  1 created  a second  questionnaire  for  African- 
American  students  where  they  answered  questions  about  “Hispanics”  on  the  assumption  that  the  stereotypes  applied  generally  to 
“others”  more  than  only  to  African-Americans.  By  including  that  data  I was  using  an  instrument  which  had  not  been  validated  for 
such  use  and  more  research  on  measurement  instruments  is  needed. 

10.  So  that  students  would  not  think  that  participation  was  in  any  way  connected  to  their  grades,  a subtle  form  of  coercion  that  is  not 
acceptable  for  the  study  of  human  subjects,  each  student  was  given  a set  of  labels  with  a 4-digit  identification  code  which  they  placed 
on  each  instrument  and  used  as  identification  at  the  beginning  of  the  software  program.  For  the  experimental  design,  this  also  meant 
that  no  data  could  be  collected  before  or  after  the  day  of  software  use  since  it  would  mean  that  I,  as  their  teacher,  knew  which  data 
was  theirs. 

11.  The  log  was  used  to  measure  total  minutes  working  with  the  software,  minutes  spent  on  genetics  questions,  number  of  pages 
viewed,  number  of  answers  entered,  number  of  correct  answers,  whether  the  student  ‘"played”  the  program  linearly  by  choosing 
sections  in  alphabetical  order,  and  the  number  of  minutes  spent  on  both  pre-  and  post-software  biology  questions. 

12.  One  example  of  this  kind  of  experiment  can  be  seen  in  the  1987  work  of  Steven  Neuberg  and  Susan  Fiske.  They  conducted  an 
experiment  where  subjects  were  told  that  they  were  going  to  work  with  a schizophrenic  ex-mental  hospital  patient  to  design  creative 
games.  The  schizophrenic  label  had  been  previously  shown  to  be  affectively  negative  so  it  was  serving  as  the  negative  stereotypic 
piece  of  categorical  information.  Some  of  the  participants  were  told  that  the  team  that  designed  the  most  games  would  win  a cash 
prize  (creating  and  manipulating  the  outcome  dependency).  Participants  were  then  given  one  of  two  “profiles”  of  the  patient  which 
contained  personal  information  (hobbies,  interests,  career  goals,  etc.).  Those  in  the  neutral  group  received  a profile  which  was  not 
consistent  with  schizophrenic  traits  while  those  in  the  dependency  outcome  group  read  a profile  that  was  positive  and  discrepant  to 
the  negative  schizophrenic  label.  This  meant  that  each  participant  in  the  study  would  have  to  choose  how  to  process  the  information 
contained  in  the  profile  which  was  inconsistent  with  the  label  of  schizophrenic  they  had  previously  been  given  about  the  patient. 
Those  in  the  dependency  outcome  group  used  individuating  impression  formation  instead  of  the  typical  category-based  evaluation, 
resulting  in  higher  evaluations  of  the  patient,  while  those  in  the  neutral  group  assessed  the  patient  using  category-based  processes,  as 
indicated  by  their  relatively  low  evaluations.  This  showed  that  if  one  is  working  with  another  toward  a common  goal,  individuating 
information  that  is  inconsistent  with  a stereotype  category  can  undermine  the  stereotype. 
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Abstract 

The  author  presents  the  methods  and  results  of  a year-long  evaluation  study  conducted  solely  for  the 
purpose  of  determining  disability  accessibility  barriers  and  potential  solutions  for  those  barriers  found  in 
four  Web-based  learning  environments.  The  methods  outlined  include  computer-based  analysis  tools  and 
computer-facilitated  focus  groups  and  focussed  individual  interviews.  Additionally , the  paper  includes 
URLs  for  several  Web  resources  on  accessibility , including  a site  created  by  the  author  based  on  the  results 
of  the  evaluation  study. 

Introduction 

The  recent  push  toward  Web-based  distance  education  has  brought  with  it  a promise  of  “anywhere  and 
anytime”  and  often  also  including  a companion  promise  of  being  “for  anyone”  However,  the  fast-paced  coding 
environment  of  the  Web  has  the  potential  to  exclude  the  population  that  may  best  benefit  from  Web-based  distance 
education,  students  with  disabilities,  thus  tarnishing  the  “for  anyone”  promise  of  these  distance  education 
technologies  and  sparking  legal  debates  centered  around  the  Americans  with  Disabilities  Act’s  “Reasonable 
Accommodations’^  for  classroom  materials  (Hinn,  1999).  With  approximately  one  in  five  people  in  the  United 
States  alone  having  a legally  defined  disability  (Waters,  1997),  the  opportunity  for  educational  outreach  is  great. 

The  idea  about  creating  accessible  Web  environments  for  people  with  disabilities  is  certainly  not  a new  one. 
An  article  that  appeared  in  an  issue  of  The  Chronicle  of  Higher  Education  in  May  of  1994  tells  the  story  of  a then 
new  graphics-based  Web  browser  called  Mosaic  that  helped  cut  the  blind  off  from  the  Internet  (Wilson,  1994). 

Before  the  advent  of  Mosaic,  the  first  graphical -mode  Web  browser,  the  Internet  was  accessed  by  a text-mode 
(command- line)  interface  without  graphics.  The  barrier  faced  by  blind  users  was  the  extreme  difficulty  that  assistive 
devices,  such  as  screen  readers  that  read  aloud  on-screen  information  or  translate  the  information  into  Braille,  have 
with  being  able  to  interpret  graphics-based  information  such  as  buttons  or  pictures  - a problem  that  still  largely  exists 
today.  5 

However,  the  inclusion  of  disability  accessibility  issues  in  evaluations  of  Web-based  instructional 
environments  has  been  extremely  uncommon  despite  the  urgings  of  evaluation  theorists  such  as  Ernest  House  to 
include  the  viewpoints  of  the  disadvantaged  in  evaluations  (House,  1993,  p.  122).  With  House’s  concept  of  social 
justice  in  evaluation  in  mind,  a year-long  evaluation  (February  1997-February  1998)  of  the  accessibility  of  a variety 
of  web-based  instructional  environments  created  at  the  University  of  Illinois  at  Urbana-Champaign  was  conducted  by 
the  author  (Hinn,  1997;  Hinn,  1998a;  Hinn,  1998b;  Hinn,  1998c). 

Evaluation  Questions 

The  primary  evaluation  questions  used  to  frame  the  evaluation  were: 

• Are  there  any  features  of  the  specific  Web-based  courseware  package  (learning  environment)  that  are  difficult  to  access 
by  persons  with  disabilities? 


4 Reasonable  accommodations  are  “modifications  to  course  materials  or  activities  that  serve  to  enable  a qualified 
student  with  a disability  to  have  an  equal  opportunity  to  attain  the  same  level  of  performance  as  a similarly  qualified 
student  without  a disability.  Reasonable  accommodations  are  required  by  law  in  accordance  with  the  Americans  with 
Disabilities  Act  (ADA)  of  1990,  as  well  as  the  Rehabilitation  Act  of  1973  and  the  Civil  Rights  Act  of  1964.  In  the 
case  of  Web-based  course  materials,  this  may  mean  providing  alternative  text-only  Web  pages  or  non-Web-based 
alternatives”  (Hinn,  1999). 

5 The  difficulties  that  graphical  information  presented  (and  still  presents)  for  many  users  with  disabilities 
has  a longer  history  than  just  the  difficulties  that  were  presented  by  the  new  graphics-based  interface  used  to  access 
the  Internet.  The  introduction  of  graphical  user  interfaces  (GUIs)  in  operating  systems,  such  as  the  one  that  1984’s 
Apple  Macintosh  operating  system  used  (and  continues  to  use),  brought  with  them  a promise  of  being  easier  to  use 
and  learn.  But  GUIs  were  not  well  received  by  the  blind.  Blind  users  continued  to  use  command-line  interfaces,  such 
as  DOS,  as  blind  users  could  not  use  the  GUI  operating  system  that  the  Macintosh  relies  on  until  software  developers 
created  programs  that  could  interpret  the  on-screen  layout  for  screen  readers  and  Braille-output  devices.  Eventually, 
however,  GUI  operating  systems  would  dominate  the  market  with  the  advent  of  GUIs  for  PC  and  Unix  computers 
and  blind  users  would  find  less  and  less  software  that  would  rely  on  the  command-line  interfaces,  such  as  the  Web 
browser  Mosaic  (Wilson,  1994). 


• What  are  the  ways  in  which  accessibility  might  be  improved  for  the  Web-based  courseware  package  (learning 
environment)? 

• Are  there  any  standard  HTML  features  of  many  Web  pages  in  general  which  are  difficult  to  access  by  persons  with 
disabilities? 

• What  tools  are  available  for  checking  accessibility  for  future  revisions  of  the  Web-based  courseware  package  (learning 
environment)? 


Participants 

Eleven  university  students  with  disabilities  participated  in  a series  of  computer-assisted  focus  groups  and 
individual  interviewed  conducted  over  the  course  of  the  year.  The  disabilities  of  the  student  participants  included 
Attention  Deficit  Disorder  with  a Learning  Disability  (1),  Traumatic  Brain  Injury  (1),  Severe  Visual  Impairments  (2), 
Blindness  (1),  Cerebral  Palsy  (1),  Muscular  Dystrophy  (3),  Quadriplegia  with  limited  arm  mobility  (1),  and 
Quadriplegia  with  no  arm  mobility  (1).  During  the  computer-assisted  focus  groups  and  individual  interviews,  the 
student  participants  used  the  assistive  technologies  that  they  would  normally  use  while  accessing  the  World  Wide 
Web  along  with  the  Web  browser  of  their  choice.  The  assistive  technologies  used  by  the  students  included  Braille 
output  devices,  keyboard  versus  mouse  navigation,  screen  magnification  software,  screen  manipulation  software,  and 
screen  readers.  Web  browsers  used  included  versions  of  Lynx  (text-mode  browser),  Microsoft  Internet  Explorer,  and 
Netscape  Navigator. 

Methods 

There  were  two  primary  evaluation  methods  used  in  the  evaluation.  The  first,  a computer-based  method, 
allowed  the  evaluator  to  rapidly  analyze  the  source  code  of  the  Web-based  evaluands  for  common  accessibility 
errors.  The  second,  the  use  of  computer-facilitated  focus  groups  and  individual  interviews,  served  as  a compliment  to 
the  source  code  analysis  with  feedback  from  the  students  with  disabilities  who  participated  in  the  evaluation.  The 
following  subsections  describe  these  methods  in  more  detail. 

Computer-Based  Analysis  Tools 

An  analysis  of  the  source  code  of  the  individual  Web  pages  in  each  of  the  evaluands  was  conducted  using 
“Bobby,”  a Web-based  analysis  tool  designed  to  help  determine  page  features  that  may  be  inaccessible  for  person 
with  disabilities.  Additionally,  each  site  was  also  checked  using  Lynx,  the  text  mode  Web  browser  that  many  blind 
students  use. 

“Bobby”  (http://www.cast.org/bobby)  was  developed  at  the  Center  for  Applied  Special  Technology  (CAST) 
and  is  a free,  automated  Web  page  analysis  tool.  There  are  two  versions  of  Bobby  which  designers  and  evaluators  of 
Web-based  instructional  resources  can  utilize.  The  original  and  perhaps  most  widely  used  version  is  Web-based  and 
allows  users  to  quickly  check  a single  page  of  a Web  site  for  its  accessibility  for  persons  with  disabilities.  In  this 
Web-based  version,  a user  simply  enters  the  URL  of  the  page  into  the  text-entry  box  on  the  Bobby  homepage.  Bobby 
then  analyzes  the  HTML  source  code,  returning  a list  of  accessibility  errors  and  suggestions  to  the  user.  Additionally, 
users  can  also  specify  a browser  version  or  HTML  version  that  they  would  also  like  Bobby  to  analyze  so  that  it  alerts 
them  to  code  that  may  be  incompatible  with  that  particular  browser  or  HTML  version. 

The  second  and  newest  version  of  Bobby  is  a standalone  Java  application6.  This  is  particularly  useful  for 
Web  site  designers  as  users  can  download  the  application  and  then  test  entire  Web  sites  on  their  local  machine.  In 
addition  to  the  analysis  features  of  the  Web-based  version,  this  version  of  Bobby  provides  the  user  with  previews  of 
what  the  Web  pages  would  look  like  in  Lynx.  The  ability  to  view  page  layout  in  at  least  a Lynx  simulation  helps 
address  one  of  the  limitations  of  Bobby  which  is  that  sometimes  it  is  not  clear  how  pronounced  a particular 
accessibility  problem  might  be.  An  example  of  this  is  the  use  of  tables.  Tables  cannot  be  viewed  in  Lynx  and  when 
Lynx  encounters  a table,  it  tries  to  come  up  with  an  alternative  layout.  Sometimes  the  layout  will  allow  the  text  in  a 
table  to  be  read  in  an  acceptable  format  that  is  easily  readable  and  understood.  But  many  times  the  layout  is  easily 
readable,  particularly  in  data  tables  such  as  ones  similar  to  a spreadsheet  worksheet.  Being  able  to  view  these  pages 
in  Lynx  is  an  advantage  for  evaluators  and  designers  alike  as  the  Lynx  previews  give  a visual  representation  for 
particularly  problematic  Web  sites  that  they  can  share  with  the  evaluation  clients. 

However,  if  an  evaluator  does  not  have  access  to  the  entire  site  or  it’s  not  convenient  to  download  the  pages 
onto  their  local  drives,  there  are  a variety  of  alternative  ways  to  view  a site  in  Lynx.  There  are  several  Web-based 
Lynx  simulations  that  allow  a user  to  enter  a URL  in  a text-entry  box  and  the  page  script  will  return  a mock-up  of 
how  the  page  would  be  viewed  to  a Lynx  user.  Two  of  these  simulations  include  “Lynx-It”  available  from  Salt  Lake 
Community  College  (http://www.slcc.edu/webguide/lynxit.html ) and  “Lynx-Me”  available  from  the  University  of 
Alberta  (http://ugweb.cs.ualberta.ca/-gerald/lynx-me.cgi).  The  most  ideal  way  of  viewing  a site  in  Lynx  is  to  actually 
go  through  the  Web  site  being  evaluated  using  the  actual  Lynx  Web  browser,  as  was  done  in  this  evaluation  study. 


6 At  the  writing  of  this  paper,  the  Java  version  of  Bobby  was  in  its  third  phase  of  beta  testing. 
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The  advantage  to  this  is  that  by  working  through  to  site,  an  evaluator  is  alerted  to  features  of  the  site  that  may  not 
work  correctly  using  Lynx.  In  this  particular  evaluation,  occasionally  a password  protection  dialog  box  or  a Web- 
based  form  used  in  an  online  quiz  did  not  work  correctly  when  used  with  Lynx.  If  these  features  had  been  viewed 
using  a Lynx  simulation,  these  difficulties  would  not  have  been  evident.  Lynx  is  available  from  several  public  telnet 
sites  including  one  at  the  University  of  North  Carolina  (telnet://public.sunsite.unc.edu  - use  “lynx’*  as  the  login)  and 
one  in  Maryland  (telnet://sailor.lib.md.us/  - use  “guest”  as  the  login). 

Computer-Facilitated  Focus  Groups  and  Individual  Focussed  Interviews 

Several  computer-facilitated  focus  groups  and  individual  focussed  interviews  with  the  student  participants 
with  disabilities  were  conducted  during  the  evaluation.  There  were  several  purposes  for  conducting  these  interviews. 
The  first  was  to  complement  the  computer-based  analysis  tools  as  a way  to  provide  richer,  more  personalized 
examples  of  accessibility  issues.  A second  purpose  of  the  interviews  was  to  help  address  the  limitation  of  the 
“Bobby”  tool  in  particular.  Since  “Bobby”  is  a source  code  analysis  tool,  there  are  many  aspects  of  accessibility  that 
cannot  be  automatically  checked  such  as  whether  or  not  a background  image  and  text  color  choices  that  are  used  for 
a Web  page  provide  a sufficient  contrast  for  test  readability.  Finally,  the  interviews  included  student  participants  with 
a wide  range  of  disabilities  that  helped  to  offset  the  primary  emphasis  of  the  computer-based  analysis  tools  on 
persons  with  visual  disabilities. 

The  computer-facilitated  focus  group  is  similar  to  the  traditional  focus  groups  in  that  they  involve  face-to- 
face  interaction  that  builds  on  group  discussion  (Worthen,  Sanders,  & Fitzgerald,  1998,  p.  382).7  However,  the 
computer-facilitated  focus  group  differs  in  that  each  focus  group  member  uses  a computer  throughout  the  focus 
group  interview.  The  advantage  to  conducting  a focus  group  in  this  manner  is  that  participants  can  directly 
demonstrate  difficulties  with  a particular  Web-based  instructional  environment  and  it  also  serves  as  a way  to  help 
ensure  that  each  participant  understands  which  particular  feature  another  focus  group  member  or  the  interviewer  is 
discussing.  In  this  evaluation,  each  participant  used  a computer  that  they  were  comfortable  with  and  that  was 
equipped  with  any  assistive  technologies  that  they  needed  in  order  to  access  Web-based  learning  environments. 

Another  difference  between  traditional  and  computer-facilitated  focus  groups  is  the  size  of  the  groups. 
Richard  Krueger  suggests  in  the  second  edition  of  his  “Focus  Groups”  (1994)  text  that  large  focus  groups  of  10  to  12 
participants  are  no  longer  seen  as  necessary,  especially  with  complex  topics.  He  suggests  that  focus  groups  with  5 to 
7 participants  are  easier  to  set  up  and  manage,  as  well  as  they  allow  individuals  to  have  more  opportunities  to  speak 
(p.  ix).  However,  computer-facilitated  focus  groups  may  require  groups  of  no  more  than  5 participants.  As  previously 
mentioned,  one  of  the  features  of  computer-facilitated  focus  groups  is  that  they  allow  the  participant  to  directly  point 
out  various  strengths  and  limitations  of  the  evaluand  to  the  evaluator,  an  advantage  that  often  requires  a greater 
amount  of  attention  by  the  evaluator  and  fellow  focus  group  members.  With  participants  with  disabilities, 
particularly  students  who  have  speech  impairments  and/or  require  a greater  amount  of  time  to  access  particular 
features  on  a Web  page  with  their  assistive  technologies,  conducting  smaller  focus  groups  can  definitely  be  more 
manageable.  There  is  a tendency  for  participants  to  move  ahead  of  others  when  working  with  Web-based  evaluands, 
especially  when  there  is  a participant  that  requires  more  time  to  navigate  through  the  site  than  the  others,  creating  a 
potential  for  chaos  in  a larger  group.  So  an  advantage  of  conducting  small  focus  groups  is  that  the  evaluator  can 
concentrate  more  fully  on  each  of  the  participants,  as  well  as  reduce  the  potential  for  confusion. 

However,  a few  of  the  participants,  were  unable  to  participate  in  even  in  a small  (2-3  person)  computer- 
facilitated  focus  group.  This  was  due  to  highly  individualized  assistive  technology  that  these  participants  required  to 
access  to  Web  pages  of  the  evaluands,  technologies  that  were  more  specialized  than  in  the  assistive  computer  labs  on 
the  University  campus.  In  these  cases,  individual  focussed  interviews  that  followed  the  same  interview  guide  used  in 
the  focus  groups  were  conducted.  A clear  limitation  in  conducting  individual  focussed  interviews  versus  focus 
groups  is  the  lack  of  idea  sharing  amongst  participants.  But  as  one  student  participant  in  this  evaluation  pointed  out, 
it  that  it  doesn’t  make  a lot  of  sense  to  conduct  a computer-facilitated  focus  group  when  the  participants  can’t  use  the 
computers  in  the  lab.  By  using  computers  and  assistive  technologies  that  a student  is  familiar  with,  the  evaluator  can 
be  more  certain  that  a difficult  to  access  Web  page  feature  is  due  to  an  access  issue  alone  rather  than  an  access  issue 
and  the  student’s  use  of  unfamiliar,  inappropriate,  or  no  assistive  technologies.  It  should  be  noted  that  the  cost  of 
constantly  updating  assistive  technologies  sometimes  prohibit  absolute  certainty  that  either  the  student  at  their 
personal  workstation  or  a group  of  students  in  an  assistive  computer  lab  is  using  the  most  ideal  assistive  technologies 
to  access  Web-based  learning  environments. 

There  are  a number  of  benefits  to  conducting  participant  individual  interviews  and  focus  groups  facilitated 
by  participant  use  of  the  Web-based  courseware  tool.  First,  observing  the  student(s)  as  they  move  through  each  of  the 
Web  pages  and  their  corresponding  features  can  reveal  strengths  and  limitation  of  the  environment  that  may  not  have 
been  addressed  in  the  original  interview  guide.  These  observations  can  also  help  shed  light  on  how  well  certain 


7 For  a more  in  depth  description  of  traditional  focus  groups,  please  refer  to  Krueger’s  book  entitled  “Focus  Groups” 
(1994). 
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assistive  technologies  that  the  student(s)  might  be  using  to  interact  with  the  Web-based  learning  environment, 
particularly  certain  components  of  the  environment  such  as  interactive  forms  or  particular  graphics.  As  both 
participant(s)  and  interviewer  discover  certain  strengths  or  limitations  of  the  environment,  the  focus  of  the  interview 
questions  can  delve  a bit  more  deeply  into  the  heart  of  a specific  strength  or  limitation. 

For  instance,  a student  might  make  a comment  that  a particular  feature,  such  as  the  “submit”  button  for  a 
form  on  a Web  page,  does  not  seem  to  work.  After  a closer  observation  of  the  participant  trying  to  interact  with  the 
feature  in  question,  the  evaluator  can  inquire  further  about  the  issue  at  hand.  This  can  help  the  evaluator  understand 
whether  or  not  the  problem  is  due  to  user  error  of  some  sort,  unclear  instruction  on  the  part  of  the  Web-based 
environment’s  developer,  or  is  perhaps  due  to  a difficulty  that  a particular  assistive  technology  has  with  interacting 
with  the  feature  in  question.  In  the  focus  group,  the  evaluator  can  also  take  this  opportunity  to  check  with  other 
participants  as  to  whether  or  not  the  feature  in  question  is  difficult  for  them  as  well.  The  group  can  work  together  to 
try  to  determine  whether  or  not  the  access  issue  is  a global  one  (e.g.,  true  for  all  Web  pages  that  contain  the  feature) 
or  more  local  one  (e.g.,  true  for  the  feature  as  implemented  on  this  page  in  particular  but  necessarily  true  across  all 
Web  pages  containing  the  feature).  An  example  of  this  from  this  evaluation  study  would  be  a quiz  submission  button 
that  was  unique  to  one  of  the  Web-based  instructional  environments  that  at  first  appeared  to  be  a more  global  issue. 
When  examining  another  Web-based  quiz  in  a different  learning  environment  that  did  not  present  a problem  for  the 
same  focus  group  participants,  it  turned  out  that  the  problematic  submission  button  was  a problem  local  to  that 
particular  learning  environment.  Additionally,  the  uniqueness  of  a particular  access  issue  can  also  enter  the  group 
discussion  where  group  participants  can  discuss  their  feelings  about  whether  the  feature  being  discussed  is 
problematic  for  students  with  one  type  of  disability  in  particular  or  whether  it  is  also  problematic  or  even  an 
advantage  for  students  with  other  types  of  disabilities. 

As  with  this  evaluation,  it  is  sometimes  the  case  that  in  the  group  discussions,  another  student  will  have  a 
solution  or  work-around  for  the  problem  that  the  student  who  first  made  note  of  the  problem  was  not  aware  of.  This 
assistive  interaction  amongst  the  participants  is  encouraged,  as  it  is  the  opinion  of  the  author  through  observation  that 
this  kind  of  helping  environment  can  be  quite  beneficial  to  the  participants.  Similarly,  the  evaluator  is  encouraged  to 
share  his  or  her  computer  expertise  or  knowledge  of  potential  solutions  to  access  barriers  with  the  participant. 

Results  of  the  Evaluation 

The  following  sections  comprise  of  the  more  generalizable  results  from  the  study.  It  should  be  noted  that  it 
is  not  a complete  listing  of  all  accessibility  issues.  For  a more  complete  listing  of  accessibility  issues,  please  see  the 
World  Wide  Web  Consortium’s  Web  Accessibility  Initiative  homepage  (http://www.w3.org/TR/WD-WAI- 
PAGEAUTH/). 

Limitations 

There  were  several  limitations  that  were  identified  by  the  student  participants  as  well  as  the  computer-based 
analysis  tools.  The  most  common  limitations  and  suggested  solutions  are  discussed  in  the  following  subsections. 

Lack  of  Alternative  Text  for  Images,  Imagemap  Hotspots,  and  Applets 

When  alternative  text  (ALT  tag)  is  not  included  within  image  tags,  for  imagemap  hotspots,  and/or  within 
Java  applet  tags,  users  who  use  text-mode  browsers  such  as  Lynx  and/or  use  screen  readers  will  be  unable  to  know 
whether  the  item  was  important  or  just  decorative.  In  these  cases,  students  who  rely  on  these  technologies,  such  as 
students  who  are  blind  or  who  have  severe  dyslexia,  will  only  get  placeholders  such  as  [image]  or  [link]  and  no  way 
of  knowing  what  the  image  might  have  been  or  what  the  link  might  be  prior  to  following  it.  In  this  evaluation,  this 
was  the  most  common  accessibility  issue  across  all  of  the  four  evaluands.  However,  adding  alternative  text  is 
probably  the  easiest  evaluation  issue  to  fix.  The  code  to  include  alternative  text  for  each  item  appears  below: 

■ Alternative  text  for  graphics:  <img  src="image_name.jpeg"  alt=”description  of  image”> 

■ Alternative  text  for  imagemap  hotspots:  <area  shape="rect"  coords="  50,50,300, 30"  href=Tile_name.htmr 

alt="description  of  link"> 

■ Alternative  text  for  Java  applets:  <applet  code="applet„name.class"  alt="description  of  what  the  applet  is  "> 

However,  it  should  be  noted  images  and  applets  may  need  more  explanation  than  just  a title  and  brief 
description  placed  in  the  ALT  tag.  Therefore  it  was  recommended  that,  in  addition  to  providing  short  alternative  text 
descriptions,  descriptions,  in  as  much  detail  as  necessary,  of  the  function  and  interpretation  of  any  complex  graphics 
(such  as  charts  or  graphs)  and  applets  should  accompany  the  image  or  applet  in  either  nearby  text  or  in  an  accessible 
alternative  on  another  Web  page. 
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Forms  Usage 

In  each  of  the  Web-based  environment  examined  in  the  evaluation,  each  used  Web  forms  for  online  quizzes 
as  well  as  chat  boards.  However,  several  of  the  students  in  the  study  needed  to  use  an  extraordinary  amount  of 
physical  effort  in  order  to  use  these  features.  This  was  particularly  the  case  for  the  participant  who  was  blind  who 
could  not  access  a quiz  using  Lynx  and  was  able  to  use  their  screen  reader  with  a graphical -mode  browser.  When 
trying  to  answer  quiz  questions,  being  able  to  read  each  question  and  then  enter  the  answer  proved  to  be  a difficult 
proposition.  When  this  student  used  the  tab  key  to  move  from  question  to  question,  the  screen  would  often  scroll  past 
the  question.  When  the  student  tried  to  get  the  screen  to  scroll  back  in  order  to  hear  the  question,  the  placement  of 
the  cursor  within  the  answer  entry  box  was  then  lost.  The  student  then  was  required  to  guess  where  the  entry  box  was 
on  the  screen  in  order  to  submit  their  answer. 

Web  forms,  such  as  those  that  might  be  used  for  an  online  quiz  or  message  board,  should  be  designed  to 
facilitate  independent  keyboard  navigation  as  much  as  possible.  Students  who  have  disabilities  that  limit  their  arm 
mobility,  such  as  those  with  cerebral  palsy  and  muscular  dystrophy,  often  also  rely  on  keyboard  navigation.  These 
students  move  the  cursor  using  the  keyboard  accessibility  features  in  their  operating  system.  Despite  the  usefulness 
of  the  keyboard  accessibility  features,  they  still  require  quite  a bit  of  manual  dexterity  to  use  for  navigating  forms, 
especially  when  the  layout  of  a form  does  not  facilitate  moving  from  answer  entry  field  to  answer  entry  field.  It  was 
recommended  to  the  developers  of  each  site  in  the  study  that  they  test  their  quizzes  using  only  the  keyboard  to 
determine  the  ease  of  navigation  in  this  manner.  However,  because  of  the  difficulties  that  the  participant  who  was 
blind  had,  it  was  also  noted  that  alternative  means  for  quiz  submission  should  also  be  supported  when  needed  and 
course  instructors  should  be  made  aware  of  possible  accessibility  issues  with  Web-based  forms  for  course  planning 
purposes. 

Frames  Usage 

The  use  of  frames  to  organize  information  in  Web  sites  has  been  a problem  for  people  using  browsers  that 
do  not  support  frames.  For  a long  time,  people  who  used  the  text-mode  browser  Lynx  could  not  access  sites  with 
frames  at  all.  More  recent  versions  of  Lynx  now  support  the  use  of  frames  by  providing  a listing  of  the  pages  that 
comprise  the  framed  site  for  the  user  to  choose  from.  This  has  resulted  in  the  unfortunate  tendency,  as  was  evidenced 
in  a few  of  the  sites  examined  in  this  evaluation,  for  Web  designers  to  assume  that  an  alternative  non-framed  site  no 
longer  is  needed  and  omitting  the  <NO  FRAMES>  instructions  altogether  or  simply  including  remarks  instructing 
the  user  to  upgrade  to  a newer  version  of  Internet  Explorer  or  Netscape  Navigator  within  the  <NO  FRAMES>  tag.  In 
one  case,  the  designers  of  one  of  the  sites  used  in  the  evaluation  had  created  a "help"  section  for  their  instructional 
environments,  one  that  contained  eight  separate  frames  without  a <NO  FRAMES>  alternative.  When  the  blind 
participant  encountered  the  help  pages,  they  were  presented  with  the  listing  of  all  eight  pages,  each  represented  by 
the  frame  name  assigned  in  the  HTML  code  as  opposed  to  their  specific  URL.  Unfortunately,  the  designers  had  also 
not  taken  care  to  assign  a frame  name  that  might  be  intuitive  to  the  user  navigating  with  a text-mode  browser  and  was 
presented  with  eight  pages  entitled  "top,"  "middle,"  "right,"  "logo,"  et  cetera.  To  make  matters  worse,  several  of  the 
pages  were  further  broken  down  and  by  the  time  the  student  found  what  they  were  looking  for,  they  no  longer  had 
any  idea  where  they  were  due  to  the  recursive  design  of  the  frames.  Had  the  designer  included  a <NO  FRAMES> 
page  that  might  have  led  the  user  to  an  annotated  set  of  alternatives  links  or  even  used  frame  names  which  more 
intuitive  names  such  as  "navigation  bar"  and  "main  frame  window,"  much  of  this  students  frustration  would  have 
been  greatly  lessened. 

Additionally,  the  Web-based  instructional  environments  that  used  frames  in  this  study  presented  unexpected 
problems  for  the  one  participant  who  was  Quadriplegic  without  any  arm  mobility.  This  student,  as  do  many  other 
students  with  disabilities  that  affect  their  arm  mobility,  used  a speech-input  device  to  navigate  through  each  of  the 
Web-based  learning  environments.  However,  when  this  student  encountered  areas  that  utilized  frames,  they  needed 
to  try  to  force  a particular  frame  into  focus  (e.g.  activate  one  frame  over  another)  so  that  they  follow  any  of  the  links 
using  their  speech-input  device.  So  while  it  was  possible  for  this  student  to  access  and  use  the  framed  site,  it 
provided  the  student  with  a few  slow  and  tedious  extra  steps.  By  providing  users  no  only  with  a <NO  FRAMES> 
alternative  within  the  HTML  code  but  also  by  providing  them  with  a link  to  this  no  frames  alternative,  either  from 
the  top  of  one  of  the  frames  or  from  a lead  page  that  gives  the  users  a choice  in  which  version  they  wish  to  browse, 
this  problem  also  could  have  been  easily  avoided. 

Graphical  Icons 

Students  with  impaired  vision  without  total  blindness  will  often  use  a graphical-mode  browser  such  as 
Netscape  and  Internet  Explorer  but  will  raise  the  font  size  to  an  extremely  large  size,  as  was  the  case  with  several  of 
the  participants  in  this  study.  However,  there  were  many  times  where  a graphic  icon  with  text  labels  as  a part  of  the 
icon  were  used  as  a navigational  device  in  a particular  learning  environment.  Since  graphic  icons  cannot  be  enlarged 
in  the  way  that  text  can  in  a Web  browser,  these  students  were  either  unable  to  differentiate  one  icon  from  another  at 
all  or  were  forced  to  sit  at  an  extremely  close  proximity  to  the  screen  in  an  attempt  to  try  to  navigate  through  the  site. 
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On  the  other  hand,  these  navigational  devices  were  noted  as  an  advantage  by  the  student  with  a learning  disability, 
noting  that  the  graphical  cue  helped  them  more  than  a simple  text  listing  of  the  pages  contained  in  the  site.  With  this 
in  mind,  it  was  recommended  to  the  developers  of  the  sites  that  they  include  text  alternatives  for  each  graphical 
navigation  device  (such  as  a cluster  of  navigational  icons  or  an  image  map)  in  addition  to  the  graphical  navigation 
device. 

Tables  Usage 

The  use  of  tables  on  Web  pages  is  another  accessibility  issue  that  holds  an  advantage  for  some  users.  There 
are  times  when  the  ideal  method  for  visually  organizing  information  is  through  the  use  of  a table.  In  fact,  the  use  of 
tables  is  sometimes  the  best  organizational  layout  for  students  with  cognitive  disabilities.  However,  for  students  who 
require  the  use  of  screen  readers,  the  text  will  become  jumbled,  as  screen  readers  will  typically  read  across  columns 
versus  one  cell  at  a time.  Another  issue  is  for  students  with  impaired  vision  without  total  blindness.  When  these 
students  use  graphical -mode  browser  by  raising  the  font  size  in  their  browser,  the  text  in  one  table  cell  will  often  end 
up  overlapping  the  text  in  another  table  cell,  returning  a confusing  layout  as  was  noted  by  several  of  the  study's 
participants.  It  was  recommended  the  site  developers  that  they  link  to  a non-tables  version  of  the  information  found 
in  each  table. 

Browser-Specific  Code 

From  time  to  time,  code  was  included  in  one  of  the  evaluands  that  did  not  work  with  text-mode  browsers. 
One  example  of  this  was  where  a graphical  icon  was  used  for  a form  submission  button  versus  the  traditional  gray 
"submit”  button.  The  code  underlying  this  feature  did  not  work  at  all  for  the  blind  participant  who  was  using  Lynx 
and  the  code  only  worked  with  the  latest  version,  at  the  time,  of  Netscape  Navigator  and  Microsoft  Internet  Explorer. 
Unfortunately,  this  was  not  evident  until  after  the  student  had  completed  the  entire  mock  quiz  used  for  the  purpose  of 
the  student  and  the  student  was  then  unable  to  submit  the  quiz. 

Another  example  of  browser-specific  code  usage  was  when  the  same  student  tried  to  access  one  another  one 
of  the  Web-based  instructional  environments.  As  with  the  other  environments,  this  site  had  a password  entry  dialog 
box.  However,  the  coding  behind  this  particular  password  dialog  box  was  browser-specific  and  the  student  was 
unable  to  access  the  site  at  all  - a pretty  severe  access  barrier.  It  was  recommended  that  the  developers  try  to  avoid 
situations  such  as  these  in  an  actual  course  situation  by  testing  each  essential  feature  that  uses  browser-specific  code 
with  a variety  of  browsers  and  prepare  a non-browser  specific  alternative.  Additionally,  it  was  also  recommended  to 
the  developers  that  they  each  prepare  a FAQ  page  that  outlines  any  known  accessibility  issues  related  to  browser- 
specific  code  as  well  as  presents  accessible  alternatives  as  well  as  a person  to  contact  for  solutions  to  any  problems 
that  have  not  yet  been  noticed  and/or  addressed. 

Strengths 

Perhaps  the  greatest  strength  found  in  this  evaluation  was  the  simple  fact  that  the  environments  allowed  for 
the  course  materials  being  online.  Having  course  resources  available  in  electronic  format  can  serve  as  an  advantage 
for  many  students  with  mobility  and  visual  disabilities.  As  one  focus  group  member  with  Quadriplegia  stated:  “For 
me... having  things  around  on  the  Net  is  a lot  easier  to  read  materials  because  I don’t  have  any  hand  movement.  So  in 
other  words,  using  a book  and  so  forth  to  look  things  up... it’s  a lot  harder.  Whereas  having  it  on  the  computer,  I can 
just  sit  there  and  go  ahead  and  work  through  it  and  get  everything  from  the  page.  But  I wish  that  I had  a lot  more 
classes  that  had  [materials]  on  the  Net”  (Hinn,  1997).  For  some  students,  Web-based  instruction  may  be  the  only  way 
that  some  students  can  independently  access  courses  and  course-materials  - something  that  is  a powerful  reminder  of 
the  need  for  accessible  online  distance  education. 

Beyond  the  Evaluation  Study 

In  addition  to  addressing  the  needs  of  students  with  disabilities  in  this  evaluation  in  order  to  increase  the 
utility  of  the  evaluands,  the  evaluation  also  was  concerned  with  maximizing  the  dissemination  of  the  evaluation 
results  to  include  the  greater  Web  design  community.  With  this  in  mind,  a Web  site  was  created  by  the  evaluator 
called  Access.Edu  (http://lrs.ed.uiuc.edu/access)  as  a way  to  try  to  reach  out  to  Web  designers  who  may  be  looking 
for  more  information  about  the  creation  of  Web  sites  that  are  accessible  to  persons  with  disabilities.  In  addition  to 
information  about  some  of  the  more  common  access  issues  from  this  evaluation,  the  site  contains  links  and 
information  about  companies,  organizations,  and  Web  accessibility  tools.  The  site  also  contains  a Web  site  design 
tutorial  called  “Considering  User  Differences”  that  was  designed  to  help  novice  Web  page  designers  understand  the 
accessibility  issues  surrounding  a designer’s  choices  of  color  and  backgrounds  in  Web  pages  through  visual  and 
interactive  examples.  This  tutorial  also  created  by  the  evaluator,  focuses  on  users  with  reading  disabilities  such  as 
dyslexia,  color-blindness,  and  visual  impairments. 

Due  to  the  academic  setting  in  which  this  evaluation  study  was  conducted,  a luxury  existed  where  the 
evaluator  was  free  to  look  solely  at  disability  accessibility  issues  in  the  four  Web-based  learning  environments  that 
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were  the  focus  of  the  evaluation.  Certainly  most  evaluators  of  Web-based  instruction  will  not  find  themselves  in 
quite  the  same  circumstances.  However,  it  is  hoped  that  the  methods  used  and  results  of  this  evaluation  will  provide 
evaluators  with  a place  to  begin  thinking  about  the  inclusion  of  disability  access  issues,  as  well  as  access  issues  in  a 
more  general  sense,  in  their  own  evaluations  of  Web-based  instruction.  It  is  also  hoped  that  the  information  in  this 
paper  will  provide  designers  of  Web-based  instructional  environments  with  a few  ideas  for  creating  more  accessible 
Web-based  learning  environments  in  the  future. 
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Abstract 

This  study  investigated  the  effects  of  audio  and  text  density  on  the  achievement , time -in -pro gram,  and 
attitudes  of  134  undergraduates.  Data  concerning  the  subjects'  preexisting  computer  skills  and  experience, 
as  well  as  demographic  information,  were  also  collected.  The  instruction  in  visual  design  principles  was 
delivered  by  computer  and  included  numerous  illustrations.  Subjects  were  randomly  assigned  to  one  of 
three  presentation  versions  of  the  instruction:  Text  Only,  Full  Text- Full  Audio,  or  Lean  Text- Full  Audio. 
No  significant  difference  in  achievement  was  found  for  the  three  treatment  groups;  however,  significant 
differences  in  achievement  were  found  for  sex  (females  achieving  more  than  males)  and  self-ratings  of 
computer  skill  (subjects  with  higher  self-ratings  achieving  more  than  those  with  lower  self-ratings).  While 
overall  attitudes  towards  the  instruction  were  favorable,  subjects  in  the  Full  Text- Full  Audio  treatment 
responded  less  favorably  than  their  counterparts  in  the  other  treatments. 

Introduction 

Multimedia  computer-based  instruction  (CBI)  is  increasingly  used  as  an  adjunct  to  instruction  in  schools 
and  corporations  and  has  typically  incorporated  text,  graphics,  and  audio.  Audio  has  often  been  added  to  CBI  as  an 
afterthought,  to  gain  attention  and  increase  motivation.  When  audio  has  been  added  as  voiced  material,  it  has  often 
occurred  as  redundant  reading  of  screen  text.  Little  research  has  been  performed  to  provide  the  instructional  designer 
with  guidelines  for  incorporating  audio  into  CBI  to  promote  learning.  When  guidelines  do  exist,  such  as  those 
proposed  by  Barron  (1995),  they  are  frequently  general  and  not  always  based  on  research.  Indeed,  little  research 
exists  to  support  the  notion  that  adding  audio  to  CBI  can  improve  learning. 

Most  research  on  the  use  of  audio  in  instruction  has  focussed  on  increasing  learning  through  multichannel 
delivery  and  has  traditionally  investigated  the  mediums  of  television,  videotape,  35-mm  slide/tape,  and  audiocassette 
teamed  with  print.  The  underlying  assumption  has  been  that  increasing  the  number  of  channels  and  the  redundancy 
between  channels  will  increase  learning.  This  assumption  stems  from  early  research  into  multichannel 
communication.  Hartman  (1961)  summarized  and  evaluated  several  studies  that  looked  at  audio-print  and  print 
presentation  for  instruction.  He  concluded  that  redundant  audio-print  instruction  was  more  effective  than  either  audio 
or  print  alone.  This  result  was  attributed  to  the  cue  summation  theory  that  predicted  learning  would  increase  as  the 
number  of  cues  or  stimuli  are  increased. 

A competing  theory  to  cue  summation  holds  that  only  one  sensory  channel  can  access  the  brain  at  a time; 
implying  that  multichannel  communication  would  not  increase  learning  (Broadbent,  1958;  Travers,  1964).  In  support 
of  that  view,  a number  of  studies  have  found  no  advantage  for  audio-print  over  print  alone  in  a variety  of  conditions 
with  different  age  groups  (Barton  & Dwyer,  1987;  Fumham,  Gunter,  & Green,  1990;  Nugent,  1982;  Van  Mondfrans 
& Travers,  1964). 

Most  audio-text  studies  have  looked  at  the  use  of  text  redundant  audio  where  the  audio  is  word  for  word  the 
same  as  the  text  and  both  are  presented  simultaneously.  A study  by  Barron  and  Kysilka  (1993)  and  another  by 
Barron  and  Atkins  (1994)  included  text  density  as  a treatment  variable.  Neither  study  found  a difference  in 
achievement  between  full  text-no  audio  and  lean  text-full  audio  groups.  If  lean  text  with  explanatory  audio 
instruction  is  as  effective  as  full  text  with  no  audio  instruction,  the  implications  for  CBI  design  are  considerable. 
Screen  "real  estate"  is  often  at  a premium — particularly  when  simulations  or  large  labeled  illustrations  or  graphics 
are  important  for  the  instruction.  Some  concepts  are  difficult  to  explain  in  words  alone — pictures  and  illustrations 
may  be  necessary  for  clear  understanding  by  the  student. 

Studies  that  have  examined  illustrations  as  well  as  text  and  audio  are  relatively  scarce.  Two  studies  (Mayer 
& Anderson,  1991;  Mayer  & Sims,  1994)  looking  at  animations  and  narrations  for  teaching  science  concepts  have 
found  that  animations  accompanied  by  simultaneous  narrations  (audio)  increase  learning.  Mayer  and  Anderson 
(1991)  and  Mayer  and  Sims  (1994)  believe  that  dual  coding  theory  provides  an  explanation  for  how  words  and 
illustrations  can  be  linked  (Paivio,  1986).  In  Paivio’s  view,  spoken  and  written  language  are  both  classified  as  verbal 
information.  In  dual  coding,  verbal  information  is  selected  and  organized  into  verbal  representations  while  visual 
information  is  selected  and  organized  into  visual  representations.  Finally,  referential  connections  are  built  between 
the  verbal  and  visual  representations.  This  entire  process  must  occur  in  short  term  or  working  memory. 


241 


229 


Other  researchers  (Mousavi,  Low,  & Sweller,  1995;  Tindall-Ford,  Chandler,  & Sweller,  1997)  have 
indicated  that  when  integration  of  diverse  elements  is  crucial  for  learning  (as  is  often  the  case  for  scientific  or 
technical  illustrations  and  accompanying  text),  audio-text  instruction  is  effective  and  may  decrease  the  burden  on 
working  memory  that  text  only  instruction  might  impose. 

The  present  study  examined  the  effects  of  audio  and  text  density  on  achievement  in  a CBI  with  content  that 
depended  heavily  on  graphics,  illustrations,  and  visual  examples.  Data  concerning  subject  demographics,  time  in  the 
CBI,  attitude  towards  the  CBI,  and  computer  experience  and  skills  were  gathered. 

The  study  was  designed  to  investigate  the  following  research  questions: 

• Does  presentation  version  (text  only,  full  text-full  audio,  and  lean  text-full  audio)  in  a CBI  affect  student  achievement, 
time-in-program  and/or  attitude? 

• Does  computer  experience  (years  of  use,  frequency  of  use,  and  self-rated  computer  skills  and  confidence)  affect  student 
achievement  and/or  attitude  in  multimedia  CBI? 

• Does  time-in-program  affect  student  achievement  and/or  attitude? 


Method 

Subjects 

One  hundred  thirty  nine  undergraduates  (93  female,  46  male)  enrolled  in  a computer  literacy  course  at  a 
large  southwestern  university  participated  in  this  study.  The  course,  offered  by  the  College  of  Education,  fulfilled  the 
university’s  undergraduate  computer  literacy  requirement.  While  35%  of  the  subjects  were  education  majors,  the 
remainder  represented  a variety  of  majors  ranging  from  Art  to  Zoology.  The  subjects  were  well  distributed  amongst 
freshman,  sophomore,  junior,  and  senior  class  standings  (18%,  24%,  39%,  and  18%). 

Materials 

A CBI,  Sending  Your  Message:  Designing  for  Web  and  Print , served  as  the  instructional  materials  for  the 
study.  The  CBI,  developed  by  the  first  author,  consisted  of  three  topics:  typography,  design  principles,  and  color. 
Each  topic  included  information,  examples,  practice  and  feedback,  and  review.  The  CBI  was  primarily  linear  in 
nature;  however,  the  subjects  could  view  previous  screens  within  any  of  the  three  topics  and  could  choose  the 
sequence  in  which  to  view  the  topics.  While  the  subjects  had  some  sequence  choices  as  they  navigated  the  CBI,  all 
subjects  were  required  to  view  all  instructional  and  review  screens,  and  to  complete  all  practice  items  before 
proceeding  to  the  posttest. 

The  program  was  designed  in  three  presentation  versions  corresponding  to  the  three  treatment  conditions  of 
the  study.  The  first  version  was  Text  Only  with  some  music  and  sound  effects,  but  no  voiced  material,  accompanying 
the  instruction.  The  second  version  was  Full  Text-Full  Audio  with  a male  voice  reading  the  instructional  text  that 
appeared  on  the  screen  word-for-word.  It  also  contained  the  same  music  and  sound  effects  as  the  Text  Only  version. 
The  third  version  was  Lean  Text-Full  Audio  with  the  instructional  text  reduced  to  a bulleted  outline  accompanied  by 
the  audio  from  the  Full  Text-Full  Audio  version.  Again,  this  version  contained  the  same  music  and  sound  effects  as 
the  Text  Only  version.  All  three  versions  contained  the  same  graphics,  number  of  screens,  practice  items  and 
reviews.  All  instructional  screens  included  graphics  to  illustrate  the  concept  or  information  presented  in  the  text  or 
audio.  Both  audio  versions  allowed  the  subjects  to  replay  the  audio. 

Procedures 

The  subjects  were  randomly  assigned  to  the  three  presentation  versions.  Students  received  instructions  from 
the  researcher,  filled  out  a demographic/computer  experience  survey,  worked  through  the  CBI,  took  a posttest,  and 
filled  out  an  attitude  survey.  The  two  surveys  were  paper-based  while  the  posttest  was  computer-based.  All  events 
occurred  within  the  students’  normally  scheduled  110-minute  laboratory  period.  All  subjects  wore  headphones  while 
working  through  the  CBI. 

Criterion  Measures 

Achievement  was  measured  with  a 30-item  posttest  administered  by  computer  immediately  after  the 
instruction.  The  posttest  consisted  of  multiple  choice,  matching  and  short  answer  items  with  each  item  worth  0 to  5 
points  depending  upon  the  number  of  answers  required  by  each  item.  Posttest  items  were  similar  in  form  and  content 
to  the  practice  items  in  the  CBI.  Posttest  multiple  choice  and  matching  items  were  scored  by  computer,  while  short 
answer  items  were  scored  by  the  first  author.  The  posttest  KR-20  reliability  was  0.80. 

The  attitude  survey  consisted  of  nine  to  eleven  Likert  scaled  (five  point  scale  from  Strongly  Agree  to 
Strongly  Disagree)  items  and  two  open-ended  items  regarding  the  usefulness,  instructional  quality  and  feelings 
towards  the  CBI.  The  number  of  Likert  scaled  items  varied  with  the  presentation  version  of  the  CBI.  For  the  audio 
versions,  two  questions  concerning  the  value  of  the  audio  were  included.  For  the  nine  Likert  scaled  items  common  to 
all  three  CBI  versions,  the  KR-20  reliability  was  0.81. 
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Results 

Achievement 

Means  and  standard  deviations  for  the  posttest  scores  are  shown  in  Table  1.  The  means  for  the  three 
versions  were  very  similar  and  the  results  of  an  ANOVA  run  on  the  posttest  means  indicated  that  there  was  no 
significant  difference  in  scores  for  the  different  CBI  versions,  F (2,  131)  = 0.725,  p > 0.05. 

Table  I.  Mean  Posttest  Scores  and  Standard  Deviations  by  CBI  Version. 


Version 

N 

Mean  Score*  (M) 

Standard  Deviation  (SD) 

Text  Only 

47 

47.23 

6.99 

Full  Text-Full  Audio 

40 

45.75 

7.10 

Lean  Text-Full  Audio 

47 

47.30 

5.95 

All  Versions  Combined 

134 

46.81 

6.66 

*A  maximum  score  of  63  points  was  possible  on  the  posttest. 


T im  e-in-  Progra  m 

Means  and  standard  deviations  for  the  time-in-program  are  shown  in  Table  2.  A one-way  ANOVA  indicated 
that  there  was  a significant  difference  in  the  means,  F (2,  131)  = 18.50,  p < 0.05.  A post  hoc  Bonferroni  test 
identified  the  source  of  the  difference.  The  mean  for  the  Text-Only  version  was  significantly  different  than  the  means 
for  the  Full  Text-Full  Audio  and  the  Lean  Text-Full  versions  at  the  p < 0.05  level.  There  was  no  significant 
difference  for  time-in-program  between  the  Full  Text-Full  Audio  and  Lean  Text-Full  Audio  versions. 


Table  2.  Mean  Time -in- Pro  gram  and  Standard  Deviations  by  CBI  Version. 


Version 

N 

Mean  Time-in-Program 
(seconds)  (M) 

Standard  Deviation  (SD) 

Text  Only 

47 

1830 

596 

Full  Text-Full  Audio 

40 

2523 

712 

Lean  Text-Full  Audio 

47 

2515 

574 

All  Versions  Combined 

134 

2277 

703 

There  was  no  correlation  between  time- in-program  and  achievement,  F (1,  132)  = 0.447,  p > 0.05  nor  were 
any  correlation’s  between  time-in-program  and  attitude  discovered. 

Attitudes 

The  means  and  standard  deviations  for  the  attitude  responses  are  shown  in  Table  3.  A MANOVA  for 
attitude  scores  indicated  no  significant  difference  for  the  different  versions,  F (9,  246)  = 1.38,  p > 0.05.  Follow-up 
univariate  tests  were  performed  for  individual  items. 
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Table  3.  Mean  Attitude  Scores  and  Standard  Deviations  by  CBI  Version  and  Versions  Combined. 


Item 

Text  Only 

Full  Text-Full 
Audio 

Lean  Text-Full 
Audio 

AH  Versions 
Combined 

1. 

I liked  this  program. 

Mean 

3.81 

3.50 

3.83 

3.72 

SD 

1.04 

0.88 

0.80 

0.92 

2. 

I would  recommend  this  program  to 

Mean 

3.72 

3.58 

3.87 

3.73 

other  students. 

SD 

0.97 

0.96 

0.72 

0.89 

3. 

I preferred  learning  about  design  in 

Mean 

4.11 

3.65 

3.85 

3.88 

a computer  program  rather  than  in  a 

SD 

1.15 

1.25 

1.09 

1.17 

lecture  format. 

4. 

This  was  a good  way  to  learn  about 

Mean 

4.17 

3.90 

4.13 

4.07 

design. 

SD 

0.99 

0.98 

0.86 

0.94 

5. 

I learned  a lot  about  design. 

Mean 

3.89 

3.50 

3.87 

3.77. 

SD 

0.98 

0.78 

0.93 

0.92 

6. 

I plan  to  use  what  I have  learned  in 

Mean 

3.85 

3.75 

4.22 

3.95 

future  work. 

SD 

1.02 

0.74 

0.66 

0.85 

7. 

This  program  was  easy. 

Mean 

4.15 

4.03 

4.11 

4.10 

SD 

0.86 

0.86 

0.71 

0.81 

8. 

I would  like  to  learn  more  about 

Mean 

3.68 

3.48 

3.70 

3.62 

design. 

SD 

1.11 

0.91 

0.81 

0.95 

9. 

I tried  hard  to  do  well  in  this 

Mean 

3.66 

3.20 

3.72 

3.54 

program. 

SD 

1.01 

0.97 

0.91 

0.98 

10. 

The  audio  helped  me  learn  better. 

Mean 

3.18 

361 

3.41 

SD 

1.15 

1.14 

1.16 

11. 

I liked  having  the  audio. 

Mean 

3.23 

3.78 

3.52 

SD 

1.10 

1.13 

1.15 

12. 

The  amount  of  practice  was 

Mean 

1.11 

1.30 

1.51 

1.30 

SD 

0.38 

0.56 

0.82 

0.63 

13. 

The  amount  of  text  on  the  screens 

Mean 

1.19 

1.23 

1.15 

1.19 

was 

SD 

0.45 

0.42 

0.47 

0.45 

Number  of  Subjects  (N) 

47 

40 

47 

134 

Notes:  Items  1-9  were  measured  on  a five-point  scale  from  1 to  5 (Strongly  Disagree  to  Strongly  Agree). 


Items  10-11  were  not  administered  for  the  Text  Only  version. 

Items  12-13  were  measured  on  a three  point  scale  from  1 to  3 (About  Right,  Too  Much,  Too  Little) 


Univariate  analyses  revealed  that  there  were  significant  differences  for  four  items  among  the  different 
versions  with  subjects  in  the  Full  Text-Full  Audio  version  responding  more  negatively  to  three  of  these  items  than 
their  counterparts  in  another  version.  Subjects  in  the  Full  Text-Full  Audio  version  (M  = 3.75,  SD  = 0.74)  responded 
more  negatively  than  those  in  the  Lean  Text-Full  Audio  version  (M  = 4.22,  SD  = 0.66)  to  the  statement,  “I  plan  to 
use  what  I have  learned  in  future  work”,  F (2,  130)  = 3.89,  p < 0.05.  A similar  pattern  was  observed  for  the 
statement,  “I  tried  hard  to  do  well  in  this  program”,  with  subjects  in  the  Full  Text-Full  Audio  version  (M  = 3.20,  SD 
= 0.97)  responding  more  negatively  than  those  in  the  Lean  Text-Full  Audio  version  (M  = 3.72,  SD  = 0.91),  F (2, 

130)  = 3.64,  p < 0.05.  In  terms  of  the  amount  of  practice,  subjects  in  the  Lean  Text-Full  Audio  version  (M  = 1.51, 

SD  = 0.82)  were  less  inclined  to  respond  that  ‘The  amount  of  practice  was  about  right”  than  subjects  in  the  Text 
Only  version  (M  = 1.11,  SD  = 0.38),  F (2,  129)  = 5.049,  p < 0.05.  Finally,  subjects  in  the  Full  Text-Full  Audio 
version  (M  = 3.23,  SD  = 1.10)  responded  more  negatively  than  those  in  the  Lean  Text-Full  Audio  version  (M  = 3.78, 
SD  = 1.13)  to  the  statement,  “I  liked  having  the  audio”,  F (1,  84)  = 5.33,  p < 0.05. 

Overall,  the  subjects  in  the  Full  Text-Full  Audio  version  appeared  to  respond  less  favorably  to  the  CBI  if 
responses  to  items  one  to  nine  in  Table  3 were  averaged  (Text  Only  M = 3.89,  Full  Text-Full  Audio  M = 3.62,  Lean 
Text-Full  Audio  M = 3.92). 


Additional  Findings 

Information  about  the  subjects,  demographics,  as.  well  as  computer  experience,  was  collected  prior  to 
starting  the  CBI.  Several  questions  regarding  computer  experience  were  asked  including:  a self-rating  of  computer 
skills,  confidence  in  succeeding  at  computer  tasks,  hours  per  week  of  computer  use,  and  years  of  computer  use. 
These  variables  and  two  demographic  variables  (Age  and  Sex)  were  examined  in  conjunction  with  achievement. 

Sex  (r  = -0.229,  p < 0.05),  self-rating  of  computer  skills  (r  = 0.252,  p < 0.05),  hours  per  week  of  computer 
use  (r  = 0.186,  p < 0.05),  years  of  computer  use  (r  = 0.210,  p < 0.05),  and  self-rating  of  confidence  in  succeeding  at 
computer  tasks  all  showed  some  correlation  with  achievement.  Not  surprisingly,  self-rating  of  computer  skills  was 
moderately  correlated  to  confidence,  hours  per  week  of  computer  use,  and  years  of  computer  use.  In  summary, 
females  achieved  significantly  more  than  males  and  subjects  with  higher  self-rating  of  computer  skills  achieved 
significantly  more  than  those  with  lower  self-ratings  of  computer  skills. 

The  status  variables  were  also  examined  in  conjunction  with  the  attitude  items.  There  was  a significant 
correlation  between  sex  and  attitudes  toward  amount  of  practice  with  females  being  more  inclined  towards  stating 
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that  there  was  enough  practice,  F (1,130)  = 5.732,/?  < 0.05  than  males.  Sex  also  correlated  with  the  statement,  “I 
tried  hard  to  do  well  in  this  program”,  F (1,131)  = 5.570,  p < 0.05  with  females  indicating  more  effort  than  males. 
The  more  confident  subjects  felt  when  dealing  with  computers,  the  more  likely  they  were  to  agree  with  the  statement, 
‘This  program  was  easy”,  F (1,131)  = 6.060,  p < 0.05.  Age  correlated  with  the  statement,  “I  would  like  to  learn  more 
about  design”,  F (1,131)  = 5.297,  p < 0.05  with  older  subjects  more  apt  to  rate  the  item  higher  than  younger  subjects. 
For  the  audio  versions,  self-rating  of  computer  skills  correlated  with  the  statement,  ‘The  audio  helped  me  learn 
better”,  F (1,84)  = 6.277,  p < 0.05  with  more  skilled  subjects  responding  more  favorably  to  the  audio. 

Discussion 

The  present  study  examined  the  effects  of  varying  levels  of  audio  and  text  density  on  achievement,  time-in- 
program,  and  attitudes  in  a CBI.  The  computer  experience,  skills,  and  demographics  of  the  subjects  were  also 
examined.  The  CBI  was  designed  following  instructional  design  principles  and  the  content  required  extensive  use  of 
graphics  for  illustrations  and  visual  examples.  All  instructional  screens  contained  some  graphic  material  that 
illustrated  the  concept  presented  in  the  text  or  audio. 

Achievement 

The  CBI  was  fairly  successful  at  teaching  the  subjects  about  basic  design,  a topic  which  few  had  any 
preexisting  knowledge.  While  the  majority  of  the  instructional  objectives  were  knowledge  and  comprehension 
objectives,  a few  were  analyze  and  evaluate.  The  number  of  analyze  and  evaluate  items  on  the  posttest  were  too  few 
to  allow  for  interpretation  in  a broader  context  but  subjects  appeared  to  score  about  as  well  on  those  as  on  the 
knowledge  and  comprehension  items. 

There  were  no  significant  differences  in  achievement  based  on  the  presentation  version  of  the  CBI.  Text 
Only  was  as  effective  as  Full  Text-Full  Audio,  a result  that  echoes  many  other  studies  (Barron  & Kysilka,  1993; 
Barton  & Dwyer,  1987;  Fumham,  Gunter,  and  Green,  1990;  Nugent,  1982;  Rehaag  & Szabo,  1995).  Again  similar  to 
other  studies  (Barron  & Aikens,  1994;  Barron  & Kysilka,  1993)  employing  CBI,  the  lean  text  version  was  as 
effective  as  a full  text  version.  By  shifting  part  of  the  verbal  portion  of  instruction  to  the  audio  channel,  more  screen 
“real  estate”  may  be  devoted  to  detailed  graphics,  illustrations,  visual  examples  and  simulations. 

The  present  results  do  not  appear  to  provide  evidence  for  the  cue  summation  theory,  as  there  was  no 
difference  in  achievement  between  the  treatments.  Some  might  argue  that  the  present  results  provide  support  for  the 
theory  that  only  one  sensory  channel  has  access  to  the  brain  at  one  time  (Broadbent,  1958;  Travers,  1964);  thus 
predicting  no  difference  in  achievement  for  the  three  presentation  versions.  Dual  coding  theory  would  view  the  text 
and  audio  as  equivalent  (verbal  information)  and  processed  simultaneously  with  the  graphics  (visual  information).  In 
this  view,  the  three  treatments  are  equivalent  and  similar  results  would  be  expected  for  each.  Research  examining 
graphics,  audio  and  text  (Mayer  and  Anderson,  1991;  Mayer  & Sims,  1994;  Tindall-Ford,  Chandler,  & Sweller, 

1997)  might  have  predicted  that  the  Lean  Text-Full  Audio  version  would  do  better  than  the  Text  Only  version  and 
possibly  better  than  the  Full  Text-Full  Audio  version. 

T ime -in- Pro  gram 

Not  unexpectedly,  the  Text  Only  version  took  much  less  instructional  time  than  either  of  the  audio 
versions — amounting  to  a difference  of  about  10  minutes  (total  instructional  time  was  30  minutes  for  the  Text  Only 
version  and  40  minutes  for  the  Audio  versions).  While  subjects  could  have  advanced  through  the  program  without 
waiting  for  the  audio  to  finish,  the  vast  majority  of  subjects  waited  for  the  completion  of  the  audio  segment  before 
continuing  to  the  next  screen.  As  time-in-program  was  more  likely  a function  of  the  presence  or  absence  of  audio,  it 
is  not  surprising  no  correlation  between  time  and  achievement  was  discovered.  Interestingly,  time-in-program  was 
not  correlated  with  attitude.  It  might  have  been  expected  that  longer  instructional  time  might  have  engendered  some 
degree  of  negative  attitude  but  such  was  not  the  case.  The  present  study  clearly  indicates  that  significant  amounts  of 
audio  in  CBI  will  substantially  increase  the  amount  of  time  required  for  instruction. 

Attitudes 

While  subjects’  general  attitudes  were  favorable  (M  = 3.81  averaged  over  all  subjects,  all  treatments  for 
items  1-9  in  Table  3),  subjects  in  the  Full  Text-Full  Audio  version  were  less  enthused  than  their  counterparts  in  the 
other  versions  (Text  Only  M = 3.89,  Full  Text-Full  Audio  M = 3.62,  Lean  Text-Full  Audio  M = 3.92).  In  fact,  the 
mean  response  for  the  Full  Text-Full  Audio  version  was  the  least  positive  of  the  three  treatments  on  each  of  the  first 
nine  attitude  items.  The  means  for  the  Text  Only  and  Lean  Text-Full  Audio  versions  were  quite  similar  for  seven  of 
the  nine  items  with  the  Lean  Text-Full  Audio  usually  being  the  more  positive  of  the  two. 
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Additional  Findings 
Implications 

The  present  study  could  have  implications  for  the  instructional  design  of  CBI  where  screen  "real  estate"  is 
needed  for  something  other  than  instructional  text.  This  is  particularly  true  for  simulations  and  concepts  difficult  to 
explain  with  text  alone.  Science  and  technical  material  as  well  as  visual  arts  and  design  are  particularly  well  served 
by  detailed  illustrations,  visual  examples,  animations  and  simulations.  If  the  screen  can  contain  a graphic  or  labeled 
illustration  with  a minimum  of  explanatory  text  while  audio  supplies  the  detailed  explanation,  a major  design 
problem  can  be  overcome. 

The  current  findings  also  provide  support  for  avoiding  redundant  audio  in  most  situations  as  an  unnecessary 
expense  that  adds  considerable  instructional  time  for  no  apparent  learning  gain.  For  some  audiences,  redundant  audio 
may  even  decrease  positive  attitudes  towards  the  instruction.  Redundant  audio  should  probably  be  reserved  for 
situations  where  the  audience  includes  poor  readers  or  used  in  brief  segments  to  emphasize  critical  points  or  issue 
warnings. 

Gen  era  lizability 

Caution  should  be  exercised  before  generalizing  results  from  this  study.  While  the  present  findings  support 
previous  research  concerning  the  effectiveness  of  Lean  Text-Full  Audio  (Barron  & Atkins,  1994;  Barron  & Kysilka, 
1993),  much  is  left  unresolved.  Both  of  the  Barron  studies  involved  technical  material  with  mostly  comprehension 
and  knowledge  objectives.  The  present  study  involved  non-technical  material  and  included  a few  analyze  and 
evaluate  objectives.  Three  studies  with  two  content  areas  do  not  constitute  much  evidence.  More  research  needs  to 
be  done  on  Lean  Text-Full  Audio  in  conjunction  with  higher  level  objectives  and  diverse  content  areas  before 
guidelines  for  instructional  designers  can  be  promulgated  with  any  certainty.  Shifting  instructional  text  to  audio  may 
only  be  effective  for  certain  types  of  content  and  objectives.  Combinations  of  content  and  objectives  need  to  be 
explored  in  greater  detail. 
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Where  is  my  brain?:  Distributed  cognition,  activity  theory,  and  cognitive  tools 
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Abstract 

This  paper  provides  a general  overview  of  distributed  cognition  and  activity  theory  in  order  to  provide 
perspective  for  a discussion  of  the  computer  as  a cognitive  tool.  Both  distributed  cognition  and  activity 
theory  enlarge  the  scope  of  “cognition”  to  include  the  individual's  interactions  with  the  environment. 
Activity  theory  offers  a framework  congruent  with  distributed  cognition  that  addresses  some  of  its 
concerns.  Despite  the  complexity  of  its  vocabulary  and  concepts , it  offers  the  potential  for  increased 
interdisciplinary  exploration  of  issues  related  to  cognitive  tools. 

Introduction 

Though  not  in  total  alignment,  a number  of  current  learning  theories  focus  on  the  interaction  of  the 
individual  with  the  environment  in  gaining  and  using  knowledge,  drawing  on  concepts  related  to  those  of  Vygostsky, 
Dewey,  and  the  Gestalt  psychologists.  Examples  include  activity  theory,  situated  cognition,  and  distributed  cognition 
(Carroll,  1997;  Cole  & Engestrom,  1993;  Hutchins,  1995;  Nardi,  1996;  Prawat,  1996).  These  theories  are  of 
increasing  interest  as  computers  become  an  integral  part  of  the  environment  in  which  knowledge  is  gained  and  used. 
This  paper  will  provide  a general  overview  of  distributed  cognition  and  activity  theory  in  order  to  provide 
perspective  for  a discussion  of  the  computer  as  a cognitive  tool. 


Situated  cognition,  distributed  cognition,  and  activity  theory 

The  terms  “situated  cognition/*  “distributed  cognition/’  and  “activity  theory’’  share  similar  concepts,  and 
sometimes  overlap  in  use.  Distinctions  are  not  always  clear.  All  move  from  studying  the  individual  in  relative 
isolation,  to  studying  the  larger  systems,  in  which  people  live  and  work.  In  addition  to  their  focus  on  interactions 
between  the  individual  and  the  community,  a distinguishing  characteristic  that  these  theoretical  perspectives  have  in 
common  is  their  acknowledgement  that  a portion  of  the  cognitive  power  used  by  an  individual  resides  in  artifacts  or 
tools  created  by  the  larger  society.  These  tools  may  be  physical,  or  they  may  be  technologies  or  concepts,  or  a 
combination. 

Situated  cognition  theories  (and  related  concepts  such  as  cognitive  apprenticeships)  stress  that  individual 
development  occurs  in  a realistic  environment,  and  is  highly  contextual  (Brown,  Collins,  & Duguid,  1989;  Collins, 
1988;  Lave  & Wegner,  1991;  Wilson,  1993).  Criticisms  of  this  theory  focus  on  charges  of  over-estimating  the 
specificity  of  knowledge,  failing  to  account  for  the  individual’s  ability  to  transfer  from  context  to  context,  and  failing 
to  consider  the  effects  of  “persistent  structures’’  and  their  design  (Anderson,  Lynn,  & Simon,  1996;  Nardi,  1996b). 

In  distributed  cognition,  the  unit  of  analysis  extends  beyond  the  individual  to  include  the  context  within 
which  the  individual  operates.  Further,  cognition  is  seen  as  less  a function  of  the  individual  as  much  as  a function  of 
an  overall  system  that  includes  groups  of  individuals  and  their  tools  (Hutchins,  1995).  Cognition  may  distributed 
socially,  across  groups  of  people,  or  may  be  distributed  across  artifacts  and  tools,  or  both.  Thus,  proponents  of 
distributed  cognition  acknowledge  that  situational  factors  such  as  social  processes  not  only  affect  individual 
cognition,  but  can  be  treated  as  integral  to  the  cognitive  process  (Hutchins,  1995;  Lajoie  & Derry,  1993;  Salomon, 
1993a). 

Moving  the  unit  of  analysis  from  the  individual  to  the  larger  system  raises  uncomfortable  questions  about 
the  role  of  the  individual  relative  to  the  system,  particularly  when  talking  about  computers  (Pea,  1993;  Perkins,  1993; 
Salomon,  1993b).  Distributed  cognition  has  been  criticized  for  emphasizing  system  goals  at  the  expense  of 
individual  goals,  or,  in  Nardi’ s terms,  assuming  a “conceptually  equivalent’’  relationship  between  people  and  artifacts 
(Nardi,  1996b).  However,  many  proponents  of  distributed  cognition  go  to  great  lengths  to  avoid  such  an  equivalence. 
For  example,  Salomon  uses  the  terms  “analytic”  and  “systemic”  to  draw  distinctions  between  the  abilities  of  the 
individual  and  the  system,  and  Pea  eschews  the  term  “distributed  cognition”  in  favor  of  “distributed  intelligence” 
because,  in  his  words,  “ people ...  ‘do’  cognition.”  (Pea,  1993;  Salomon,  1990;  Salomon,  1993b;  Salomon,  Perkins,  & 
Globerson,  1991). 

Activity  theory,  a development  of  sociocultural  theory,  offers  a flexible  framework  that  addresses  many  of 
these  concerns,  one  that  some  authors  already  see  as  merging  with  or  even  subsuming  the  concept  of  distributed 
cognition  (Carroll,  1997;  Nardi,  1996b).  Activity  theory  specifically  states  that  relations  between  individuals  and 
artifacts  are  not  symmetrical;  artifacts  may  be  mediators  of  human  thought  and  behavior,  but  human  motive  and 
consciousness  belong  to  people,  not  things  (Kaptelinin,  1996b).  Conceptually,  distributed  cognition  and  activity 
theory  are  very  closely  related;  the  same  collection  of  articles  will  often  include  both  perspectives  (see,  for  example, 
Salomon,  1993a).  In  many  cases,  ideas  from  distributed  cognition  and  from  activity  theory  seem  indistinguishable: 
Pea  (1993),  writing  under  the  banner  of  distributed  cognition,  addresses  important  themes  that  are  conceptually 
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equivalent  to  those  in  activity  theory.  Such  contributions  are  of  great  value,  and  should  not  be  overlooked.  However, 
activity  theory  offers  a broader  and  more  developed  theoretical  framework  in  which  to  examine  questions  related  to 
the  distribution  of  cognition. 

Activity  theory 

The  underlying  concepts  of  activity  theory  originate  in  the  work  of  Leont’ev,  a follower  of  Vygotsky 
(Leont'ev,  1978).  Confusingly,  “activity  theory”  now  refers  to  both  the  original  Soviet  psychology  tradition  and  to 
more  recently  developed  theories  that  extend  and  develop  the  original  concepts  (Cole  & Engestrom,  1993;  Kuutti, 
1996).  Current  contributors  are  international  and  interdisciplinary,  including  representatives  from  psychology, 
anthropology,  human -computer  interaction  (HCI),  cultural  research,  and  other  social  sciences.  The  history  and 
development  of  activity  theory  is  beyond  the  scope  of  this  paper,  which  will  sketch  only  a few  key  concepts,  but 
numerous  references  offer  more  comprehensive  discussions  (for  example,  Cole,  1995;  Cole,  Engestrom,  & Vasquez, 
1997;  Nardi,  1996a). 

The  activity  system 

Activity  theory  provides  a powerful  descriptive  framework  focused  around  a “mediated  activity  system,” 
which  “comprises  the  individual  practitioner,  the  colleagues  and  co-workers  of  the  workplace  community,  the 
conceptual  and  practical  tools,  and  the  shared  objects  as  a unified  dynamic  whole”  (Engestrom,  1992).  Figure  1 
depicts  a general  activity  system  as  described  by  Engestrom. 

Figure  1.  Model  of  an  activity  system,  adapted  from  Engestrom  (1992). 


Mediating  artifact  (tool) 


labor  (roles) 


Carroll  (1997)  describes  activity  theory  as  follows: 

The  object  of  description  in  this  approach  is  an  “activity  system,”  the  ensemble  of  technological  factors  with  social 
factors,  and  of  individual  attitudes,  experiences  and  actions  with  community  practices,  traditions,  and  values. 
Activity  theory  emphasizes  that  these  ensembles  are  inherently  contingent  and  changing,  that  human  activities  are 
mediated  and  transformed  by  human  creations,  such  as  technologies,  and  that  people  make  themselves  through 
their  use  of  tools....  Activity  theory  shifts  attention  from  characterizing  static  and  individual  competencies  toward 
characterizing  how  people  can  negotiate  with  the  social  and  technological  environment  to  solve  problems  and 
learn,  which  subsumes  many  of  the  issues  of  situated  and  distributed  cognition,  (p.  512) 

In  Engestrom’s  model  of  an  activity  system,  the  components  of  the  system  continually  influence  and 
transform  one  another;  the  lines  in  the  figure  are  sometimes  even  drawn  as  double-headed  arrows,  to  reflect  such 
mutual  influences.  At  the  same  time,  each  system  is  a node  in  a network  of  related  activity  systems;  for  example, 
tools  in  an  activity  systems  are  products  of  previous  activity  systems  for  tool  production  (Engestrom,  1992).  Thus, 
current  activity  systems  are  influenced  by  layers  of  historical  development. 

The  vocabulary  related  to  activity  theory  is  complex,  and  requires  some  patience  during  initial  contact.  A 
few  key  concepts  relating  to  activity  theory  are  needed  for  further  discussion;  however,  this  overview  should  be 
considered  as  only  a preliminary  introduction  to  the  topic. 
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Activities , actions , and  operations 

Kuuti  (1996)  provides  an  excellent  description  of  the  hierarchical  levels  of  an  activity.  At  the  highest  level 
is  an  activity , which  is  essentially  defined  at  the  level  of  motives  or  goals.  “An  activity  is  a form  of  doing  directed  to 
an  object,  and  activities  are  distinguished  from  one  another  according  to  their  objects.  Transforming  the  object  into 
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an  outcome  motivates  the  existence  of  an  activity”  (Kuutti,  1996,  p.  27).  Objects  are  an  important  and  difficult 
construct  in  activity  theory,  since  the  term  is  used  both  in  the  sense  of  “objective,”  as  in  purpose  or  intention,  and  in 
the  sense  of  the  “thing”  to  be  transformed  (Kuutti,  1996;  Nardi,  1996b).  Kuutti  (1996)  observes  that  the  original 
Russian  term  for  activity  carries  the  connotation  of  ‘doing  in  order  to  transform  something,’  which  he  considers 
essential  (p.41).  Thus,  while  “objective”  or  goal  is  probably  the  easiest  English  translation  for  beginners,  it  is  not 
complete. 

Objects  can  and  do  evolve  and  change  during  an  activity  system,  but  are  relatively  slow  to  change.  Actions 
are  shorter-term  processes  underlying  activities,  which  accomplish  goals  related  to  the  activity.  Actions  require  the 
context  of  an  activity  to  be  understood.  An  activity  can  be  achieved  through  a variety  of  actions,  and  a single  action 
can  belong  to  multiple  activities.  The  example  Kuutti  offers  is  the  action  of  making  a workplace  project  report, 
which  might  belong  to  a “project  management”  activity  or  to  a “competing  for  promotion”  activity. 

Below  the  level  of  actions  are  operations , “well-defined  habitual  routines  used  as  answers  to  conditions 
faced  during  the  performance  of  an  action.”  (Kuutti,  1996,  p.  31).  Kaptelinin  offers  this  distinction  between  the  three 
levels: 

The  criteria  for  separating  these  processes  are  whether  the  object  to  which  the  given  process  is  oriented  is 
impelling  in  itself  or  is  auxiliary  (this  criterion  differentiates  between  activities  and  actions),  and  whether  the  given 
process  is  automatized  (this  criterion  differentiates  between  actions  and  operations).  (Kaptelinin,  1996a,  p.  108-109). 

However,  actions  can  be  folded  into  operations  as  they  become  more  familiar,  and  operations  can  return  to 
the  level  of  actions  in  response  to  changing  conditions  that  demand  greater  attention.  The  boundaries  between 
actions  and  activities  are  equally  fluid,  and  movement  is  possible  in  both  directions. 

Functional  organs 

Another  key  concept  concerns  functional  organs , or  goal-directed  combinations  of  human  abilities  with 
tools  that  improve  existing  capabilities,  or  even  allow  new  functions  (Kaptelinin,  1996a;  Kaptelinin,  1996b; 
Zinchenko,  1996).  These  combinations  can  be  seen  as  belonging  to  the  individual,  much  as  the  stick  of  Bateson’s 
blind  man  can  be  considered  as  an  extension  of  his  senses,  and  can  be  quite  temporary  in  nature  (Cole  & Engestrom, 
1993;  Kaptelinin,  1996b). 

Some  functional  organs  extend  the  internal  plane  of  action  (IP  A),  or  the  capability  for  mentally 
representing  and  transforming  external  objects  (Zinchenko,  1996).  This  is  an  important  and  complex  concept: 
activity  theory  explicitly  attempts  to  describe  interactions  between  internal  consciousness  and  the  external  world  in  a 
non-dualistic  manner.  “Context  is  both  internal  to  people  - involving  specific  objects  and  goals  - and,  at  the  same 
time,  external  to  people,  involving  artifacts,  other  people,  specific  settings.  The  crucial  point  is  that  in  activity  theory, 
external  and  internal  are  fused,  unified...”  (Nardi,  1996b,  p.76). 

Contradictions 

Because  activities  are  affected  by  related  activities,  tensions  are  inevitable.  The  terms  contradiction  and 
breakdown  refer  to  misfits  between  different  activities,  or  even  among  elements  in  an  activity  system.  Such 
contradictions  are  sources  of  tension,  but  can  also  be  sources  of  development  (Cole  & Engestrom,  1993;  Kaptelinin, 
1996b).  According  to  Cole  (1993)  “Activity  systems  are  best  viewed  as  complex  formations  in  which  equilibrium  is 
an  exception  and  tensions,  disturbances,  and  local  innovations  are  the  rule  and  the  engine  of  change.”  (p.  8). 
Explicitly  identifying  contradictions  within  a system  can  lead  to  positive  changes  (for  case  studies,  see  Cole  & 
Engestrom,  1993;  Engestrom,  1992;  Favorin  & Kuutti,  1996). 

Artifacts  as  mediators  of  human  action 

Artifacts  - tools,  symbols,  and  signs  - are  the  result  of  previous  activity  systems.  They  incorporate 
“crystallized”  knowledge  and  implicit  goals,  which  can  shape  the  goals  of  their  users  (Kaptelinin,  1996b).  At  the 
same  time,  users  actively  create  the  meaning  of  a tool  as  they  interact  with  it  and  the  environment,  and  do  not  merely 
react  to  “affordances”  (Kaptelinin,  1996b). 

An  interesting  perspective  is  that  artifacts  are  created  by  people  to  provide  external  support  and  control. 
“Artifacts,  broadly  defined  to  include  instruments,  signs,  languages  and  machines,  mediate  activity  and  are  created 
by  people  to  control  their  own  behavior.  Artifacts  carry  with  them  a particular  culture  and  history  (Kuuti  1991)  and 
are  persistent  structures  that  stretch  across  activities  through  time  and  space.”  (Nardi,  1996b,  p.  75.  Emphasis 
added.)  Carroll’s  (1996)  description  of  activity  theory  includes  the  phrase  “people  make  themselves  through  their  use 
of  tools”  (p.  512). 

These  perspectives  on  artifacts  will  contribute  to  the  following  discussion  of  the  computer  as  a cognitive 

tool. 
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Current  areas  of  development 

Activity  theory  combines  a number  of  the  elements  of  concern  to  educational  technologists  into  a powerful 
descriptive  framework.  In  the  United  States,  two  groups  have  been  especially  influential  in  extending  and  refining 
activity  theory.  At  the  Laboratory  of  Comparative  Human  Cognition  in  San  Diego,  Cole  and  Engestrom  form  the 
center  of  a group  who  concentrate  on  the  cultural  contexts  of  learning  and  work,  juxtaposing  activity  theory  with  the 
work  of  Jean  Lave,  James  Wertsch,  Dewey,  and  various  sociocultural  anthropologists  (Cole  et  al.,  1997).  Another 
core  has  developed  within  the  field  of  human-computer  interaction  (HCI).  This  group,  which  includes  Nardi,  Kuutti, 
and  Kaptelinin,  sees  activity  theory  as  providing  a contextual  framework  for  the  consideration  of  artifacts  (especially 
computers)  that  connects  to  previous  HCI  research  directions  (Carroll,  1997;  Nardi,  1996a).  In  their  explorations  of 
activity  theory,  these  groups  draw  on  overlapping  research  traditions  and  integrate  previous  research  with  this 
developing  descriptive  framework. 

Applying  activity  theory  to  educational  technology 

Many  current  concerns  of  educational  theory,  particularly  as  related  to  computers,  can  be  discussed  using 
activity  theory,  which  offers  a flexible  framework  that  addresses  the  dynamic  and  complex  nature  of  educational 
interactions.  For  example,  the  description  of  activities,  actions,  and  operations  can  clearly  be  related  both  to 
motivational  theories  and  to  cognitive  theories  about  “chunking.”  The  active  role  of  the  individual  corresponds  to  a 
constructivist  viewpoint,  while  the  inclusion  of  both  individual  and  community  goals  incorporates  the  social 
dimension  of  knowledge  and  culture.  The  conceptualization  of  artifacts  encompasses  theories  about  “affordances,” 
while  acknowledging  both  the  situated  nature  of  artifacts-in-use  and  the  learned  responses  of  the  individual  user. 
Though  researchers  may  be  interested  in  exploring  specific  relationships  within  an  activity  system,  describing  these 
interactions  as  part  of  an  overall  system  maintains  an  awareness  of  contextual  factors  - the  “forest”  within  which  the 
“trees”  are  rooted. 

While  the  descriptive  power  of  activity  theory  can  most  easily  be  brought  to  bear  on  specific  cases  and 
situations,  applying  the  framework  to  describe  larger  issues  can  also  be  profitable.  Here  is  an  example  of  how  the 
vocabulary  of  activity  theory  can  enrich  understanding  some  of  the  issues  surrounding  computers  as  cognitive  tools. 

Using  activity  theory  to  describe  the  computer  as  a cognitive  tool 

Jonassen  and  Reeves  (1993)  offer  the  following  definition  of  cognitive  tools:  “Cognitive  tools  refer  to 
technologies,  tangible  or  intangible,  that  enhance  the  cognitive  powers  of  human  beings  during  thinking,  problem 
solving,  and  learning.  Written  language,  mathematical  notation,  and  most  recently,  the  universal  computer  are 
examples  of  cognitive  tools.”  (p.  693). 

Many  authors  distinguish  using  the  computer  as  a cognitive  tool  from  using  it  as  a medium  for  the  delivery 
of  instruction,  often  linking  to  constructivist  learning  principles  in  which  the  learner  designs  with  the  tool.  (Cole  & 
Engestrom,  1993;  Jonassen  & Reeves,  1996;  Lajoie,  1993;  Pea,  1993;  Pea,  1985;  Perkins,  1993;  Salomon,  1993a; 
Salomon,  1993b;  Salomon,  1993c;  Salomon  et  al.,  1991).  However,  “enhancing  cognitive  powers”  can  be  interpreted 
in  multiple  ways,  and  affect  what  one  considers  a cognitive  tool.  At  one  extreme,  some  limit  the  term  to  tools  that 
intentionally  develop  human  capability,  however  that  development  is  accomplished.  At  the  other  extreme,  tools  that 
augment  human  performance  (and  perhaps  make  some  learning  unnecessary)  are  included.  In  between  are  a range  of 
programs  such  as  microworlds,  intelligent  tutoring  systems,  expert  systems,  and  the  now-commonplace  computer 
applications  usually  shelved  under  “productivity  programs”  - for  example,  spreadsheets,  databases,  and  word- 
processing programs. 

One  of  the  discussions  associated  with  cognitive  tools  is  their  effects  on  the  person  using  the  tool;  a 
particular  concern  is  the  inadvertent  “de-skilling”  of  the  user.  In  the  context  of  distributed  cognition,  Salomon  has 
provided  a critical  distinction  between  effects  with  and  effects  of  the  computer  (Salomon,  1990).  Effects  with  the 
computer  refer  to  the  efficacy  of  the  total  system  (the  person-plus-computer).  In  contrast,  effects  of  the  computer 
refer  to  the  “cognitive  residue”  left  upon  the  person,  changes  evident  when  operating  as  a solo  performer.  Salomon 
also  proposes  evaluating  human-computer  partnerships  according  to  systemic  and  analytic  criteria.  The  former  looks 
at  the  total  performance  (effects  with),  while  the  latter  specifies  the  specific  contributions  of  user  and  system.  Pea 
(1993)  acknowledges  the  distinction  between  analytic  and  systemic  criteria,  but  takes  issue  with  the  assumption  that 
a low  proportion  of  user  contribution  is  cause  for  alarm.  He  persuasively  argues  Salomon’s  approach  is  overly 
simplistic,  and  fails  to  consider  the  evolving  relationships  between  the  users  and  tools. 

Looking  at  this  discussion  through  the  filter  of  activity  theory  clarifies  the  two  separate  kinds  of  interaction 
under  consideration.  One  involves  the  role  of  the  computer.  There  are  goals  inherent  in  the  design  of  the  computer 
program  in  question,  which  is  the  outcome  of  a previous  activity  system.  One  way  these  goals  are  evidenced  is  in  the 
division  of  labor  between  the  individual  and  the  computer  in  the  current  activity  system,  or  the  role  that  the  computer 
plays  in  the  system  (See  Figure  2).  The  dark  solid  line  indicates  that  interaction. 
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Figure  2.  Describing  computers  as  cognitive  tools  within  an  activity  system. 
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The  dotted  line  indicates  the  second  area  of  consideration,  which  are  the  interactions  and  transformations 
between  the  individual  and  the  program.  These  include  both  the  “cognitive  residue”  upon  the  person  and  the 
individual's  adaptation  of  the  tool  to  his  or  her  own  goals  (possibly  in  ways  unforeseen  by  the  developers).  Like  all 
interactions  in  an  activity  system,  these  two  interactions  are  mutually  influenced  by  one  another,  and  by  the  other 
interactions  within  the  system,  but  focusing  on  them  separately  for  the  moment  allows  goals  and  concerns  to  be  more 
clearly  identified  for  discussion. 

Distinguishing  between  these  two  types  of  interactions  helps  to  clarify  the  issues  surrounding  cognitive 
tools,  and  identify  some  of  the  points  of  tension  (contradictions  or  breakdowns)  within  the  discussion. 

The  following  synthesis  draws  on  the  literature  specifically  discussing  cognitive  tools  (for  example, 

Jonassen  & Reeves,  1996;  Pea,  1985;  Perkins,  1993;  Salomon,  1990;  Salomon,  1993a;  Salomon,  1993b;  Salomon, 
1993c),  from  the  literature  on  electronic  performance  support  systems  (EPSS’s),  a type  of  cognitive  tool  most 
commonly  used  in  business  (for  discussion,  see  Collis  & Verwijs,  1995;  Dorsey,  Goodrum,  & Schwen,  1993;  Gery, 
1995;  Laffey,  1995;  Raybould,  1995;  Rosenberg,  1995;  Scales  & Yang,  1993),  and  from  considerations  of  computer- 
based  learning  environments  (Cognition  and  Technology  Group  at  Vanderbilt  University,  1991;  Hannafin,  1992; 
Lajoie,  1993;  Land  & Hannafin,  1996). 

In  Table  1,  the  first  column  addresses  the  division  of  labor  commonly  discussed  in  relation  to  computers  as 
cognitive  tools.  Of  course,  this  division  could  be  conceptualized  in  multiple  ways.  In  this  analysis,  the  division  is 
considered  not  in  terms  of  specific  tasks,  but  in  terms  of  the  role  that  the  computer  plays  in  respect  to  the  user: 
expert,  intellectual  partner,  or  servant.  Such  roles  are  of  course  fluid,  and  may  be  operative  at  different  points  during 
the  use  of  the  same  program. 
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Table  I.  Considering  the  “ division  of  labor”  between  computers  and  single  users  as  evidenced  in  computer 
roles.  


Potential  influences  upon  the  individual 

COMPUTER  ROLES 

goals 

Concerns 

questions 

(desirable  outcomes) 

(undesirable  outcomes) 

(contradictions) 

• teaches  user,  improving  solo 

• allows  successful  outcomes 

• What  are  the  competencies 

i 

k 

performance 

without  learning 

and  skills  needed  in  today’s 

society? 

• creates  “zone  of  proximal 

• encourages  over-reliance  on 

development’* 

“authority”  of  the  computer. 

• What  balance  should  exist 

between  process  and 

• scaffolds  (as  in  cognitive 

• fails  to  develop  necessary  skills 

product?  How  do  we 

apprenticeships) 

• develops  “useless”  skills 

evaluate  the  two? 

• identifies  processes  and  strategies 

• How  should  skills  for 

• encourages  emphasis  of  form 

• provides  tools  for  knowledge 

over  content 

monitoring,  assessment, 
and  reflection  be 

representation 

• fails  to  develop  reflective  skills 

developed? 

P 

artner 

• provides  feedback  on  decisions 

' 

• extends  capabilities  for  error 

• What  information, 

• provides  situated  learning 

knowledge,  or  skills  ought 

experiences  in  the  context  of 

• obscures  critical  decision  points 

people  have  “in  their 

performance 

• reduces  human  abilities 

heads,”  as  solo  performers? 

Why? 

• extends  capabilities  in  a synergistic 

• reduces  “executive  control”  of 

relationship 

tasks 

• Can  software  design  ensure 

that  users  perceive  critical 

• allows  focus  on  strengths 

• reduces  command  of  necessary 

information  and  decision 

1 

f 

• allows  off-loading  of  onerous 

information 

points? 

mental  tasks 

Servant 

• reduces  need  to  learn  unnecessary 

information 

In  the  “servant”  role,  the  computer  primarily  supports  lower-level  cognitive  processes  in  order  to  free  the 
user  to  engage  in  higher  level  activities,  or  augments  memory.  Users  may  off-load  tedious  or  repetitive  tasks,  or  refer 
to  stored  information.  They  may  also  create  reminders  or  similar  low-level  procedural  supports. 

As  the  computer  offers  increased  informational  access,  procedural  support,  and  representational 
capabilities,  its  role  is  more  comparable  to  a partner  or  expert.  The  distinction  between  these  roles  can  be  subtle,  and 
may  have  less  to  do  with  the  capabilities  of  the  tool  than  with  the  nature  of  the  interactions  between  the  user  and  the 
computer  - specifically,  how  much  of  the  “goal  direction”  is  assumed  by  the  computer,  and  how  much  is  consciously 
retained  by  the  user.  In  an  expert  system,  the  intelligence  built  into  the  computer  program  is  responsible  for 
decisions,  and  the  user  relies  upon  it  for  guidance.  Expert  systems  can  encourage  learning  by  modeling  problem- 
solving strategies  or  working  methods,  or  they  may  operate  in  a “black  box”  fashion. 

As  a partner,  the  user  and  the  computer  work  in  an  interactive  partnership,  directed  by  the  user's  intent.  The 
computer  not  only  amplifies  but  re-organizes  mental  functioning  (Pea,  1985;  Perkins,  1985;  Perkins,  1993).  In  this 
role,  the  computer  most  approaches  being  a functional  organ , increasing  the  ability  to  represent  problems  and  to 
envision  the  results  of  potential  decisions. 

In  the  second  column  of  Table  1,  the  goals  and  concerns  of  developers  and  researchers  concerning  potential 
influences  on  the  individual,  or  “cognitive  residues,”  are  grouped,  roughly  corresponding  to  the  role  of  the  computer. 
The  division  into  desirable  and  undesirable  categories  reflects  the  general  discussion  in  the  literature,  though  many 
items  could  be  debated.  These  goals  and  concerns  are  obviously  shaped  by  historical  influences,  including  the  values 
of  the  educational  community. 

From  this  simple  organizational  strategy,  some  interesting  questions  can  be  developed,  which  are  presented 
in  the  final  column  of  the  table.  These  questions  are  complex,  but  by  no  means  exhaustive.  They  merely  illustrate  the 
kinds  of  underlying  tensions  that  shape  the  development  and  assessment  of  cognitive  tools  - potential  contradictions, 
in  activity  theory  terminology. 

Looking  at  this  portion  of  this  particular  type  of  activity  system  is  only  the  beginning.  A more 
comprehensive  examination  of  this  same  interaction  would  consider  not  only  the  computer’s  impact  on  the  user,  but 
the  user's  impact  on  the  computer , which  might  include  creating  a knowledge  base  or  constructing  specialized  tools 
(Laffey,  1995;  Lesgold,  1993).  Other  aspects  of  the  activity  system  should  also  be  considered,  and  patterns 
identified:  for  example,  in  discussions  of  EPSS’s  and  other  workplace  computer  systems,  contradictions  between 
community  objectives  (the  goals  of  the  organization  implementing  the  system)  and  individual  objectives  (the  goals  of 
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a typical  end-user)  are  gaining  increasing  attention  (Carroll,  1997;  Kling  & Jewett,  1994;  Scales  & Yang,  1993). 

Alan  Cooper,  speaking  from  decades  of  experience  as  a software  developer  and  interface  designer,  offers  a trenchant 
description  of  a typical  contrast: 

The  user’s  goals  are  often  very  different  from  what  we  might  guess  them  to  be.  For  example,  we  might  think 
that  an  accounting  clerk’s  goal  is  to  process  invoices  efficiently.  This  is  probably  not  true.  Efficient  invoice 
processing  is  more  likely  the  goal  of  the  company  or  the  clerk’s  boss.  The  clerk  is  more  likely  concentrating  on  goals 
like: 

■ Not  looking  stupid 

■ Not  making  any  big  mistakes 

■ Getting  an  adequate  amount  of  work  done 

■ Having  fun  (or  at  least  not  being  too  bored). 

If  you  think  about  it,  those  are  pretty  common  goals . (Cooper,  1995,  p.  12). 

They  are  pretty  common  goals,  and  seem  to  be  as  prevalent  in  education  as  in  business.  Perkins  (1985) 
points  out  that  an  individual  goal  of  many  student  users  of  computers  seems  to  be  “avoiding  cognitive  load,”  citing 
programming  examples  involving  Logo.  Obviously,  there  is  a contradiction  between  this  goal  and  the  objectives  of 
the  educational  community! 

Areas  for  additional  research 

Writing  about  cognitive  tools  for  pedagogy,  Salomon  points  out  that 

No  tool  is  good  or  bad  in  and  of  itself;  its  effectiveness  results  from  and  contributes  to  the  whole  configuration  of 
events,  activities,  contexts,  interpersonal  processes  taking  place  in  the  context  of  which  is  used.... If  nothing 
significant  changes  in  the  classroom  save  the  introduction  of  a tool,  few  if  any  important  changes  can  be  expected. 
(Salomon,  1993c,  p.  189) 

Research  confirms  that  tools  like  Logo  can  have  widely  different  effects  when  used  in  varying  contexts. 
(Jonassen  & Reeves,  1996;  Pea,  Kurland,  & Hawkins,  1987).  Salomon  (1994)  points  out  in  regard  to  media  and 
cognition  studies  that  there  is  often  a great  difference  between  what  can  be  affected  and  what  is  typically  affected; 
the  same  is  true  for  any  technology,  whether  the  potential  is  for  good  or  ill  (Salomon,  1994).  Using  perspectives  from 
activity  theory  to  revisit  the  use  of  such  tools  may  help  to  identify  contextual  factors  that  influence  the  widely 
varying  results  that  have  been  reported. 

The  previous  example  used  concepts  from  activity  theory  at  a fairly  abstract  level,  adopting  the  framework 
to  summarize  an  ongoing  discussion;  a more  common  and  practical  approach  is  to  use  activity  theory  to  analyze 
specific  systems.  While  activity  theory  is  descriptive,  not  prescriptive,  it  has  been  used  for  both  analysis  and 
diagnosis  (B0dker,  1996;  Cole  & Engestrom,  1993;  Engestrom,  1992;  Engestrom  & Escalante,  1996;  Favorin  & 
Kuutti,  1996).  The  results  of  such  an  analysis  can  even  be  used  to  consciously  re-design  the  activity  system,  as  Cole 
and  Engestrom  (1993)  demonstrate  in  two  different  examples  (though  neither  specifically  focused  on  computers). 

However,  what  constitutes  a “better”  activity  system  is  always  open  to  debate:  the  answers  involve 
individual  and  societal  goals  as  much  as  they  involve  capabilities  of  people  and  tools.  Considering  the  activity 
system  as  a whole  - and  acknowledging  the  value-laden  questions  that  arise  in  considering  “objects”  - is  one  way  to 
address  concerns  about  the  potential  consequences  for  individuals  and  society  as  computers  become  more  commonly 
used  as  cognitive  tools. 

Summary 

Theoretical  perspectives  provide  both  organizing  structures  and  a set  of  filters.  Both  distributed  cognition 
and  activity  theory  enlarge  the  scope  of  “cognition”  to  include  the  individual’s  interactions  with  the  environment. 
Activity  theory  offers  a framework  congruent  with  distributed  cognition  that  addresses  some  of  its  concerns.  This 
framework  has  a rich  history,  and  is  being  extensively  developed  by  disciplines  related  to  educational  technology, 
such  as  psychology  and  human-computer  interaction.  Despite  the  complexity  of  its  vocabulary  and  concepts,  it  offers 
the  potential  for  increased  interdisciplinary  exploration  of  issues  related  to  cognitive  tools. 
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Abstract 

This  qualitative  study  examined  the  structure  and  interaction  in  an  online  course  from  the  meaning  - 
perspectives  of  the  actors  involved.  Data  collected  included  interviews , observations  and  messages  posted 
by  participants  to  a virtual  classroom  environment.  Findings  suggested  that  while  teachers  conceded  that 
they  spent  more  time  teaching  an  online  course  than  a traditional  face-to-face  course , they  believed  the 
online-teaching  experience  to  be  a worthwhile  endeavor.  In  the  context  of  an  online  course , teachers  felt 
that  they  were  more  'present9  and  available  for  students  than  in  a traditional  course. 

Results  of  the  study  also  suggest  that  students  displayed  better  commitment  to  their  work  in  an  online 
course  than  a traditional  course  because  they  have  close  interaction  with  the  teacher.  This  close 
interaction  can  offset  the  lack  of  control  the  students  may  feel  in  a highly  structured  distance  course.  The 
personality , emotions , philosophy  and  educational  background  of  the  teacher  determine  the  structure  and 
design  of  an  online  course ; including  the  amount  of  interaction , and  time  the  teacher  spends  on  online 
activities. 

Introduction 

Most  early  definitions  of  distance  education  include  a teacher  and  a student  separated  by  geographical 
distance  or  time  (Mclsaac  & Gunawardena,  1996).  The  learner,  teacher,  and  a medium  of  communication  are  the 
most  distinguishing  features  of  distance  education.  Interactions  and  relationships  between  teachers  and  students  in 
distance  education  are  extremely  important  and  learners  usually  carry  on  dialog  with  instructors  across  space  and 
time,  using  communications  media.  Students  work  within  specified  course  requirements  or  structure  determined  by 
the  teacher  in  advance.  The  geographic  distance  between  learner  and  teacher  creates  what  is  termed  'transactional 
distance' (Moore,  1991). 

Course  structure  and  learner-teacher  dialog  are  important  elements  in  transactional  distance.  Transactional 
distance  is  defined  by  Moore  (1991)  as  a “...[distance  of  understandings  and  perceptions,  caused  in  part  by  the 
geographic  distance,  that  has  to  be  overcome  by  teachers,  learners  and  educational  organizations  if  effective, 
deliberate,  planned  learning  is  to  occur.”  Moore  postulated  that  high  (rigid  and  inflexible)  program  structure  reduced 
dialog,  and  therefore  increased  transactional  distance.  Several  researchers  including  Saba  and  Shearer  (1994), 
Bischoff,  Bisconer,  Kooker  and  Woods  (1996)  have  verified  and  expanded  the  concept  of  transactional  distance. 

They  elaborated  on  Moore's  work  by  saying  that  transactional  distance  decreases  when  dialog  increases  and  structure 
increases.  Conversely,  increased  structure  and  decreased  dialogue  increase  transactional  distance.  Students  in  a 
distance  setting  communicate  more  with  their  teacher  than  students  in  a traditional  class  format  (Bischoff  et  al, 

1996).  Communication  in  distance  settings  is  usually  through  the  medium  of  electronic  mail;  however,  several 
courses  now  employ  highly  interactive  technologies  such  as  two-way  video. 


Interaction  between  Teacher  and  Student 

From  the  student's  perspective,  interaction  with  the  teacher  is  a means  of  obtaining  feedback  and  guidance. 
Feedback  in  an  online  environment  may  be  either  synchronous  (immediate  and  in  real-time)  or  asynchronous 
(delayed).  Timely  response  to  questions  from  students  produced  relatively  good  results  in  online  courses  (Starr, 
1995).  This  was  irrespective  of  the  type  of  course  and  the  kinds  of  online  activities  required  by  the  teacher.  Other 
research  findings  (Egan,  et  al.,  1991)  on  the  need  for  interaction  have  shown  that  learners  value  timely  feedback 
regarding  course  assignments,  exams,  and  projects.  Instructors  comment  that  students  in  an  online  environment 
expect  more  immediate  feedback  than  students  in  a traditional  class.  While  increased  teacher- student  interaction  is 
good,  it  is  time-consuming;  therefore,  it  is  important  to  examine  whether  increased  interaction  with  the  teacher 
affects  the  quality  of  student  learning.  While  studies  (Harasim,  1990;  Waggoner,  1992)  show  that  computer  mediated 
conferencing  with  student- teacher  interaction  produces  positive  results,  this  is  an  issue  that  warrants  more  research 
(Kearsley,  1995). 

Beginner  teachers  of  online  courses  feel  overwhelmed  with  the  amount  of  time  they  spend  interacting  with 
each  student  (Tinker,  1997).  The  model  that  is  used  in  designing  a particular  course  is  key  to  determining  the  time 
spent  on  that  course.  Some  teachers  expect  to  communicate  with  every  student  individually  as  they  try  to  replicate 
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the  traditional  model  of  instruction  in  online  teaching.  Such  a model  can  make  communication  with  even  a small 
group  of  students  tedious,  and  can  make  professors  feel  as  if  they  are  holding  unlimited  office  hours.  A less 
cumbersome  and  more  productive  model  is  the  seminar  model  (Tinker,  1997)  in  which  the  teacher  structures  the 
virtual  classroom  by  laying  out  activities  and  facilitating  communication  among  students.  In  this  model  the  teacher 
refrains  from  having  extended  individual  conversations  with  students.  Instead,  the  teacher  encourages  students  to 
communicate  with  each  other,  moderates,  and  guides  discussions.  The  advantage  of  such  a model  is  that  it  is  more 
scalable  and  productive  for  the  teacher.  Also,  it  is  more  suited  for  constructivist  learning  environments,  and  allows 
the  student  to  actively  construct  an  internal  representation  of  knowledge  by  interacting  with  the  material  to  be 
learned.  Even  when  implementing  a seminar  model,  a teacher  can  spend  a lot  of  time  on  an  online  course  when 
compared  with  a traditional  face-to-face  class. 

Time  Spent  By  Teaching  Faculty 

While  students  can  approach  a teacher  directly  during  class  in  a traditional  classroom  when  they  have 
problems,  students  in  an  online  course  can  participate  and  receive  feedback  on  a daily  basis.  The  onus  is  therefore 
on  the  instructor  to  sign  on  to  the  conferencing  system  on  a daily  basis  and  respond  to  any  waiting  questions.  Starr 
(1995)  reports  that  teachers  believe  that  it  takes  them  about  30  minutes  to  an  hour  each  day  to  complete  this  activity 
depending  on  the  number  of  students  and  the  extent  to  which  they  are  involved  in  the  course.  The  conscientious 
teacher  therefore  becomes  a "perpetual  professor"  (Starr,  1995)  rather  than  teaching  for  a few  specific  hours  in  a 
week.  Teachers  need  to  keep  abreast  of  ongoing  dialog  in  different  discussion  tracks  and  maintain  class  web  sites 
(University  of  Wisconsin,  1996;  Creed,  1996).  While  learners  benefit  significantly  from,  and  are  more  motivated  if 
they  are  in  frequent  contact  with  the  instructor  (Coldeway,  et  al.,  1980);  teachers  find  this  a very  time-intensive  task. 
Kearsley  (1995)  also  urges  distance  education  programs  to  make  cost/benefit  evaluations  of  interaction  in  terms  of 
preparation  time  versus  instructional  effects.  In  a case  study  that  focused  on  a distance  education  class  taught  using 
television,  he  concludes  that  the  benefit  of  increased  interaction  are  higher  student  involvement  and  satisfaction  with 
courses;  however,  he  is  unconvinced  that  the  costs  involved  in  obtaining  these  results  balance  out  the  benefits. 

Several  researchers  have  called  for  further  research  on  teaching  an  online  course  from  the  teacher's 
perspective.  It  is  important  to  examine  the  obstacles  and  limitations  in  teaching  an  online  course  (Berge,  1997)  and 
examine  the  distance  teacher’s  perspective  of  dialog,  structure  and  transactional  distance  (Bischoff  et  al,  1996).  The 
present  study  explores  teacher-learner  interaction  from  the  perspective  of  the  teacher,  and  outlines  the  complex 
interaction  of  time  spent  on  teaching  an  online  course  in  relation  to  course  structure  and  dialog.  The  study  also 
examines  the  area  of  time  spent  by  teachers  of  online  courses  and  attempts  to  further  the  understanding  of  how  the 
time  is  spent  and  whether  teachers  consider  teaching  online  a worthwhile  investment  of  their  time.  Important 
outcomes  of  teaching  the  course  such  as  satisfaction,  and  a sense  of  achievement  are  examined  from  the  perspectives 
of  the  actors  involved  in  the  particular  setting.  It  is  important  for  us  to  examine  issues  faced  by  faculty  related  to 
time,  particularly  because  time  is  a top  priority  for  most  distance  education  faculty  (Sherry,  1996). 

Purposes  Of  the  Study 

To  explore  the  issues  of  interaction  and  time  discussed  above,  the  following  questions  were  investigated: 

1 . What  are  the  responsibilities  of  an  online  teacher? 

2.  How  much  time  does  the  teacher  spend  on  each  of  these  responsibilities? 

3.  Does  the  teacher  consider  teaching  the  online  course  a worthwhile  experience? 

4.  How  does  the  philosophy  and  background  of  the  teacher  affect  course  design  and  consequently  the  amount  of  interaction 
and  time  the  teacher  spends  on  an  online  course? 


Method 

Setting  and  Participants 

The  participants  in  this  study  were  a teacher  and  a teaching  assistant  who  taught  a graduate  course  in 
distance  education  delivered  through  the  medium  of  a computer  conferencing  system  during  summer  1998.  The 
instructor  is  a professor  who  has  been  teaching  courses  about  distance  education.  She  pioneered  the  use  of  online 
technologies  for  course  delivery  at  the  university  for  graduate-level  courses. 

The  course  was  one  of  the  several  courses  offered  by  Arizona  State  University  in  the  distance  setting 
(http://seamonkey.ed.asu.edu/~mcisaac/disted/).  There  were  seventeen  students  in  the  class  that  summer.  The  course 
employed  the  World  Wide  Web  and  computer  conferencing  technologies  for  course  delivery  and  included  a 
combination  of  face-to-face  and  online  instruction.  It  was  conducted  mainly  at  a distance  because  teachers  and 
students  did  not  meet  face-to-face  except  for  the  first  three  sessions  of  the  class.  The  class  was  supported  with  the 
help  of  a text-based  computer  conferencing  software  called  FirstClass  which  is  a closed  environment  that  only 
members  of  the  class  can  access.  The  system  allows  students  and  teachers  to  engage  in  both  synchronous  (chat)  and 
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asynchronous  (e-mail,  contribution  to  folders)  communication.  It  also  provides  file  transfer  capability  and  can  link 
directly  to  the  web.  This  feature  enables  students  to  connect  to  specific  web  addresses  that  are  included  in 
discussions. 

Several  folders  were  created  as  part  of  the  FirstClass  environment.  The  News  folder  was  available  for 
general  discussions  between  students  and  any  announcements  that  teachers  wanted  to  make.  Students  were  required 
to  participate  in  group-discussions  each  week  by  posting  messages  to  specific  folders.  The  general  area  of  discussion 
was  predetermined.  However,  students  could  choose  a specific  topic  of  discussion  within  the  given  area.  Students 
were  required  to  act  as  moderators  for  at  least  one  discussion  (either  individually  or  in  groups)  in  the  semester  and 
had  to  post  two  questions  for  each  discussion  on  the  weekly  topic.  Every  student  had  to  respond  to  each  question  at 
least  once  during  the  week.  It  was  the  moderator's  responsibility  to  summarize  the  results  of  the  discussion  and  post 
them  online  once  the  discussion  was  complete. 

Teachers  and  students  met  face-to-face  during  the  first  three  days  of  the  course.  This  was  done  so  that 
students  could  familiarize  themselves  with  the  conferencing  system  and  the  format,  requirements,  and  schedule  of  the 
course.  The  class  web  site  provided  information  about  assignments,  schedule,  resources  to  students.  The  teaching 
assistant  maintained  postings  of  students’  work  and  constantly  updated  that  portion  of  the  web  site.  After  the  first 
three  face-to-face  meetings,  class  continued  online  for  five  weeks.  Then,  class  met  face-to-face  one  more  time  at  the 
end  of  the  course. 

Data  Collection 
Qualitative  Data 

Erickson's  (1986)  interpretive  method  was  chosen  to  analyze  the  qualitative  data.  Erickson  uses  the  process 
of  analytic  induction  for  data  analysis.  His  approach  to  qualitative  data  analysis  is  holistic.  He  believes  in  looking  at 
the  data  as  a whole,  rather  than  splitting  up  the  data  into  categories.  The  researcher  did  not  code  the  data,  but  read 
and  re-read  the  data  in  order  to  generate  assertions  from  the  data.  Assertions  are  generally  statements  that  one  can 
make  about  the  data.  They  may  be  statements  about  relationships  observed  in  the  data,  hypotheses  or  propositional 
statements.  Warranting  of  assertions  is  important  in  the  Erickson  approach.  Warranting  consists  of  searching  the 
entire  data  for  confirming  and  disconfirming  evidence  for  the  assertion.  On  finding  disconfirming  evidence,  the 
researcher  may  need  to  reframe  the  assertion  or  modify  it.  If  confirming  evidence  does  not  predominate,  then  an 
assertion  may  be  dropped.  However,  sometimes,  an  assertion  may  be  acceptable  even  if  the  amount  of  confirming 
evidence  is  greater  than  the  amount  of  disconfirming  evidence  available.  Ideally,  assertions  are  warranted  by  many 
instances  of  data  from  various  sources  such  as  interviews,  field  notes  and  site  documents.  This  step  of  warranting 
assertions  is  important  because  it  ensures  that  the  researcher  is  not  reacting  to  rare  and  dramatic  events.  Finding 
these  links  between  the  data  and  the  events  are  crucial  for  Erickson.  The  interpretivist  notion  of  validity  hinges  on 
coherence.  It  is  vital  for  the  researcher  to  include  a plausible  account  of  the  construct.  We  have  attempted  to 
document  and  warrant  assertions  in  detail  in  this  study. 

Qualitative  data  were  collected  through  interview,  online  messages  and  chats.  Interviews  were  audio  taped 
and  transcribed  later  for  analysis.  Messages  that  teachers  sent  out  on  FirstClass,  and  chat  sessions  with  peers  and 
students  were  analyzed.  Teachers  participated  in  text-based  chat  sessions  with  students  and  peers,  which  were  saved 
by  them  for  analysis  on  a voluntary  basis. 

Quantitative  Data 

The  quantitative  data  that  were  collected  was  primarily  the  amount  of  time  that  teachers  spent  in  various 
activities  as  part  of  the  process  of  teaching.  Data  collection  was  based  on  self-report  because  teachers  were  teaching 
this  course  at  a distance  and  direct  observation  was  not  possible.  Teachers  participated  in  a variety  of  activities 
related  to  the  course,  such  as:  planning  and  preparation,  teaching,  administration,  interaction  with  peers,  interaction 
with  the  learning  environment  (conferencing  system)  and  interaction  with  students.  Quantitative  data  is  reported  in  a 
separate  study.  This  paper  examines  teacher-student  interaction  from  the  perspective  of  the  teacher  and  documents 
teacher  perceptions  regarding  time  spent  on  teaching  the  distance  class. 

Researcher  Role,  Choices  Made  and  Implications 

The  researchers  were  participants  in  the  class  setting.  One  of  the  researchers  was  a complete  participant 
and  the  other  researcher  occupied  a role  where  she  was  more  of  an  observer  than  a participant.  Being  participants  in 
the  setting  enabled  us  to  collect  data  that  was  rich  in  detail,  particularly  in  the  interview  mode.  Our  familiarity  with 
the  participants  made  it  possible  for  us  to  obtain  intimate  details  about  their  background  and  gain  better  access  to 
their  thoughts  during  the  interviews.  Further,  both  researchers  have  taught  courses  using  web  and  conferencing 
technologies  as  a medium.  Thus,  we  were  naturally  interested  in  this  setting,  and  able  to  easily  obtain  access  to  the 
setting. 
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As  the  researchers  observed  the  settings,  interviewed  the  actors,  and  thought  about  the  data,  several  issues 
emerged  from  the  observations  and  interviews.  Researchers  reviewed  the  data  obtained  from  the  various  sources 
several  times.  They  identified  certain  common  patterns  and  themes  that  emerged  from  the  data.  Following  are  the 
main  themes  that  most  of  the  data  related  to: 

• Teaching  style  in  the  distance  setting:  Teachers  teach  differently  in  the  distance  setting  than  they  do  in  the  traditional 
setting.  Teachers  perceive  their  role  to  be  different  in  the  online  setting  when  compared  with  their  role  in  the  traditional 
face-to-face  setting. 

• Dialog  with  students:  Dialog  is  made  up  of  different  forms  of  communication  with  students.  Examples  are  chat,  e-mail, 
and  face-to-face  interaction.  It  was  obvious  that  these  forms  of  communication  also  were  used  for  a variety  of  purposes 
and  that  what  was  communicated  is  just  as  important  as  how  it  was  communicated. 

• Time  spent  on  teaching  the  distance  class:  Teachers  felt  that  they  spent  more  time  on  teaching  a distance  class  than  a 
traditional  class.  In  fact  the  other  two  issues  outlined  here  seemed  to  be  directly  connected  to  the  time  spent  on  the  class. 

Since  rich  data  sources  were  available  for  assertions  concerning  teaching  style  and  dialog  with  students,  the 
present  study  is  focused  on  these  issues.  Based  on  repeated  readings  of  the  data  obtained  from  various  sources  such 
as  interview,  observations  and  text  messages  from  the  conferencing  system,  researchers  generated  some  assertions. 
Then  we  proceeded  to  warrant  our  assertions  by  searching  the  data  for  confirming  and  discontinuing  instances. 

These  assertions  have  changed  and  evolved  several  times  during  the  process  of  analysis.  We  found  that  these 
assertions  were  strong  because  there  was  a preponderance  of  data  that  supports  the  assertion  rather  than  data  that 
disconfirms  the  assertion.  Based  on  this  process,  some  of  our  preliminary  assertions  were  dropped,  and  two 
assertions  were  strengthened.  The  first  assertion  examines  the  issue  of  time  spent  on  teaching  online  courses  and  the 
second  assertion  explores  the  area  of  student-teacher  dialog.  We  will  now  present  these  two  and  provide  warrant  for 
them. 

Assertion  1:  While  teachers  may  be  required  to  spend  more  time  when  they  teach  a course  in  the  online 
format,  they  believe  that  it  is  worthwhile  for  them  to  put  in  the  extra  effort 

Teachers  who  teach  courses  at  a distance  perceive  the  need  to  do  a lot  of  work  to  put  a course  online.  A 
teacher  needs  additional  help  to  start  an  online  course,  and  then  continue  teaching  it  in  that  format.  Teaching  in  an 
online  or  computer  mediated  environment  seems  to  somehow  alter  the  role  of  the  teacher  as  well  as  change  the 
interaction  dynamic  between  teacher  and  students  and  between  students  themselves.  In  the  traditional  classroom,  the 
teacher  is  the  source  of  knowledge  and  people  seem  to  believe  that  knowledge  flows  from  the  teacher  to  the  student. 
In  the  computer  mediated  classroom,  learning  requires  the  student  to  participate  in  a variety  of  activities  such  as 
interaction  and  discussion  with  other  students,  interaction  with  the  teacher,  reading  messages  posted  in  the 
conferencing  system  and  accessing  the  class  web  page  for  content  and  resources.  The  teacher  believes  that  she  gets  a 
greater  level  of  commitment  with  students  who  participate  in  the  computer-mediated  environment.  She  feels  that  this 
may  primarily  be  due  to  the  fact  that  students  can  be  present  in  the  learning  environment  only  when  they  are  ready  to 
participate  and  contribute  to  the  learning  environment.  This  viewpoint  may  be  illustrated  in  the  following  quotation 
from  my  interview  with  her: 

Teacher:  And  that's  what  I like  about  the  online  courses,  is  that  when  someone  is  there,  they  are  doing  something 
or  they  are  not  there.  So  I think  it  is  a much  more  individual  sense  of  time  and  what  they  are  doing.  And  I like  that. 

I also  like  the  level  of  commitment  and  interaction  that  you  get,  which  I didn't  get  in  a traditional  class.  I would 
really  struggle  sometimes  to  get  somebody  to  contribute  something. 

The  teacher's  perception  was  that  she  got  a better  idea  of  each  individual  person's  thought  process  in  the 
online  environment  whereas  it  was  easier  for  some  students  not  to  participate  in  a traditional  teacher-centered  class. 
She  also  felt  that  the  mediated  environment  seems  to  encourage  better  interaction  because  students  do  not  have  the 
traditional  distractions  that  they  may  have  in  a face-to-face  environment.  Students  have  an  opportunity  to  really 
think  about  what  they  want  to  say  before  they  post  it  to  the  conferencing  system,  and  this  seems  to  enhance  the 
quality  of  what  they  have  to  say.  Further,  every  student  has  an  opportunity  to  contribute  to  class  discussions  in  this 
online  environment  and  they  tend  to  write  and  communicate  freely  with  the  teacher  using  the  medium  of  chat  or  e- 
mail.  The  teacher  mentioned  that  from  past  experience,  it  is  only  students  who  are  new  to  the  conferencing 
environment  who  are  usually  reluctant  to  initiate  online  conversations  with  her.  When  questioned,  they  said  that  they 
did  not  want  to  interrupt  whatever  she  was  doing.  She  mentioned  that  she  now  usually  makes  it  a point  to  initiate 
contact  with  students  via  chat  when  they  are  logged  on  simultaneously  and  let  them  know  that  she  was  available  if 
they  needed  help.  This  clarified  things  and  encouraged  them  to  communicate  with  her  if  they  needed  help. 

In  spite  of  the  fact  that  she  needs  to  put  in  extra  effort,  the  online  course  has  somehow  brought  the  teacher 
closer  to  her  students.  She  is  able  to  relate  to  the  distant  students  and  feel  close  to  them  because  she  can 
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communicate  individually  with  them  and  understand  problems  that  they  might  be  having.  In  the  online  setting,  the 
teacher  has  an  opportunity  to  interact  with  all  students  in  the  class  unlike  her  traditional  class  where  only  certain 
students  communicated  with  her.  Students  also  concurred  with  this  viewpoint.  The  following  is  an  excerpt  from  an 
evaluation  that  one  of  the  students  wrote: 

Student  1: 1 think  that  I learned  more  in  this  class  then  I would  have  in  a traditional  setting.  In  a traditional  setting, 

I would  not  have  interacted  with  the  instructors  and  students  as  much  as  I did.  Also,  I would  not  have  had  as  much 
time  to  work  on  the  assignments,  do  research  and  read,  as  I did  in  this  class.  I often  find  that  the  research  and 
reading  get  pushed  aside  when  I am  in  a traditional  course,  but  this  did  not  occur  in  this  class....  I would  also  like  to 
thank  the  instructors  for  their  timely  feedback.  I found  it  very  useful  to  be  able  to  chart  my  progress  and  to  get 
such  specific  feedback. 

Students  feel  very  involved  with  the  online  class  and  relate  to  it  in  a different  manner.  An  important  factor 
that  may  contribute  to  involvement,  is  course  structure.  Students  were  required  to  participate  in  group-discussions 
and  collaborate  with  group  members.  Requiring  students  to  take  responsibility  for  what  happens  in  the  classroom 
environment  and  providing  close  interaction  with  faculty  seems  to  foster  a sense  of  commitment  in  the  students: 

Student  2:  I also  enjoyed  collaborating  with  group  members.  I felt  more  responsible  for  my  participation  in  the 
course  and  felt  that  I “influenced”  the  course  in  some  way. 

When  teachers  participated  in  online  discussions  and  were  simply  present  online,  they  felt  that  students 
were  more  satisfied  and  participated  actively  in  the  conferencing  system.  The  students  felt  that  the  teacher  was  more 
’present’  and  available  for  them,  and  consequently,  the  teacher  was  able  to  get  more  out  of  the  students.  Both 
teachers  and  students  felt  closer  to  one  other  and  were  able  to  share  thoughts  and  ideas.  They  contend  that  while  this 
might  also  happen  in  a traditional  class  for  some  students,  all  students  in  a distance  course  would  have  the 
opportunity  to  experience  this  closeness.  Another  student  expressed  her  thoughts: 

Student  3:  There  were  many  things  I liked  about  the  course.  I liked  being  able  to  talk  to (the  teaching 

assistant)  a lot.  I found  him  very  reassuring.  He  helped  to  alleviate  my  fears.  Whenever,  I had  a problem,  he  was 
always  a chat  command  or  e-mail  away.  He  also  made  sure  not  only  to  chat  with  me  about  class-related  elements, 
but  also  about  more  personal  things  such  as  the  topic  of  his  dissertation  and  his  future  career  plans,  making  me  feel 
like  he  cared  about  more  than  my  academic  progress.  I think  this  is  very  important  for  instructors  to  do,  and  I try  to 
do  it  myself. 

The  teacher  had  a teaching  assistant  who  helped  her  immensely  in  teaching  the  course  and  in  administering 
the  conferencing  system.  This  was  an  important  factor  because  it  gave  her  the  opportunity  to  focus  on  facilitating  the 
course  and  spending  a good  amount  of  time  interacting  with  students.  The  teaching  assistant  helped  out  with  all 
aspects  of  the  course,  particularly  in  grading  assignments  and  responding  to  technology  related  among  other  things. 
The  teacher  and  the  assistant  shared  their  responsibilities,  so  that  the  assistant  focused  more  on  administrative, 
technical  functions,  while  the  teacher  maintained  a general  overview  and  conducted  most  of  the  other  activities 
related  to  the  course. 

In  analyzing  the  data  with  regard  to  time,  it  is  interesting  to  note  that  in  a traditional  course  taught  in  the 
summer,  teachers  would  have  to  spend  a total  of  about  75  to  100  hours  on  the  course.  In  this  online  course,  however, 
teachers  spent  two  to  five  times  as  much  time.  Another  interesting  pattern  noted  was  that  the  teaching  assistant  spent 
more  than  twice  the  amount  of  time  the  teacher  spent  on  the  class  (377.75  hours  versus  98.83  hours  spent  by  the 
teacher).  In  fact,  looking  at  the  data  collected  with  regard  to  the  amount  of  time  that  the  teacher  spent,  30.83  hours 
was  spent  interacting  with  students.  In  contrast,  the  teaching  assistant  spent  94.16  hours  interacting  with  students.  It 
is  obvious  that  the  teaching  assistant  spent  a greater  amount  of  time  on  the  course,  even  though  percentages  of  time 
spent  look  very  similar.  This  could  have  been  partially  due  to  the  fact  that  the  assistant  was  a younger  and  more 
inexperienced  teacher.  Thus,  he  needed  to  spend  more  time  teaching  and  preparing  for  the  class  when  compared  to 
the  experienced  teacher.  It  probably  also  had  something  to  do  with  the  personality  of  the  teaching  assistant.  The 
assistant  loved  to  work  online  and  was  logged  in  even  at  times  when  he  was  not  required  to  be.  Thus,  when  students 
logged  in,  they  always  had  an  opportunity  to  interact  with  him  via  chat  or  e-mail.  Looking  at  percentages  of  time 
spent  in  the  various  activities  pertaining  to  the  class,  it  was  interesting  to  note  that  both  teaching  assistant  and  teacher 
managed  to  constantly  remain  in  touch  with  each  other  and  the  students  even  when  they  were  located  physically  in 
different  states. 

A study  of  the  patterns  of  time  spent  indicate  that  teachers  spent  more  time  on  administration  and  planning 
than  in  the  area  of  interaction.  Further,  since  the  class  required  to  students  to  participate  in  online  discussions, 
teachers  spent  a lot  of  time  reading  these  discussions  and  contributing  to  them.  Overall,  the  online  course  required  at 
least  two  teachers  for  a group  of  17  students.  They  also  had  an  intern  who  helped  in  posting  assignments  on  the  web. 
One  reason  why  online  courses  need  so  much  time  may  be  because  of  the  fact  that  these  courses  need  to  be 
constantly  updated.  Teachers  noted  that  online  courses  need  a lot  of  work  even  if  they  had  taught  the  course  in  a 
similar  format  earlier.  Teachers  required  students  to  use  Internet  resources  for  the  course  and  they  agreed  that 

263 


best  copy  available 


250 


updating  online  resources  was  a time-intensive  task.  Interaction  with  students  and  administration  also  involved  a lot 
of  work.  In  spite  of  having  to  spend  the  extra  time  on  the  course,  teachers  like  to  teach  in  this  format.  One  important 
reason  for  this  was  flexibility.  Teachers  can  teach  the  course  from  anywhere  and  they  have  the  ability  to  reach 
students  asynchronously  at  any  time  via  e-mail.  Thus,  they  could  complete  their  responsibilities  at  their  convenience. 
Flexibility  was  an  important  factor  for  students  also,  as  indicated  in  their  course  evaluations.  Teachers  felt  that  this 
flexibility  enabled  students  and  teachers  to  experience  a better  quality  of  learning  and  teaching.  This  viewpoint  is 
evident  in  the  following  excerpt  from  an  interview  with  the  teacher: 

Teacher.  Well,  I think  it  is  much  more  time  intensive,  but  I like  the  quality  of  time  better.  1 think  that  if  I compare 
the  two,  going  to  class  once  a week,  and  having  office  hours,  I interact  with  a lot  more  students  individually.  The 
only  people  who  came  into  office  hours  were  people  who  were  on  campus.  Most  people  were  not  going  to  make  an 
effort  to  come  to  campus  to  appear  at  an  office  hour.  So,  generally,  I got  to  know  some  students  really  well  in  a 
face-to-face  classroom.  But  if  I had  to  go  back  now  and  teach  these  courses  once  a week  and  that  was  the  only 
contact  I had  with  the  students,  I wouldn’t  be  very  happy. 

Most  students  liked  the  course  once  they  got  used  to  the  technology  and  the  format.  This  is  evident  from 
another  student  evaluation: 

. Student  4:  I also  think  that  once  you  become  comfortable  with  a new  environment  it  is  difficult  to  go  back  to  the 
traditional  learning  environment  without  feeling  some  dissatisfaction.  Once  you  know  and  experience  other 
possibilities  you  tend  to  want  to  continue  to  have  those  options.  Before  the  experience  there  was  nothing  to 
compare...  I would  definitely  take  another  distance  learning  class.  In  fact,  given  a choice,  I am  sure  I would  choose 
this  format  over  the  traditional  classroom  environment. 

Teachers  and  students  feel  that  they  got  to  know  each  other  well  during  the  course  of  the  class.  This  helped 
students  build  a rapport  with  teachers  and  contributed  to  their  course  participation.  Considering  the  pros  and  cons  of 
teaching  in  an  online  environment,  teachers  perceived  that  it  was  worthwhile  to  put  in  the  extra  effort  to  teach  such  a 
course  because  it  afforded  them  flexibility  and  brought  them  closer  to  their  students.  This  closeness  consequently 
seems  to  enhance  student  participation  and  satisfaction  with  the  course  as  evaluated  based  on  course  evaluations. 

Assertion  2:  Teaching  style,  personal  philosophy  and  background  of  a teacher  affect  course  design  and 
student-teacher  interaction  in  an  online  course. 

Teachers  have  particular  teaching  styles  and  this  affects  everything  they  do  as  teachers.  Teachers  have 
various  reasons  for  adopting  a particular  style  of  teaching.  It  was  important  for  the  teacher  of  the  course  under  study 
to  get  students  involved  and  actively  engaged  in  the  course.  Her  teaching  philosophy  directly  affected  the  way  she 
designed  the  course.  Her  personal  teaching  style  influenced  by  her  educational  experience  and  background  had  a 
profound  effect  on  teaching  style. 

Teachers  have  various  reasons  for  adopting  a particular  style  of  teaching;  even  a specific  method  of  using 
technology  as  a medium  of  teaching.  The  participant  in  the  study  did  not  outline  instructional  objectives  and  select 
appropriate  learning  experiences  systematically  at  the  beginning  of  the  course;  nor  did  she  identify  a static  body  of 
information  that  she  would  transmit  to  students.  She  took  control  of  the  fact  that  she  was  having  problems  with  her 
face-to-face  class  and  decided  to  re-structure  the  class  so  that  it  was  more  student-centered.  The  following  excerpt 
from  her  interview  underscores  this  feeling: 

Teacher.  There  is  something  missing  with  my  teaching  in  face-to-face  class.  I can  go  into  a classroom  and  it  is 
really  great  and  I have  all  these  lectures  prepared,  and  I used  to  do  those  cooperative  learning  exercises... I always 
felt  that  they  were  a little  contrived.  I was  always  trying  to  plan  how  to  involve  people  and  get  them  going  and  at 
the  end  of  it,  I felt  good  about  the  class,  but  I could  see  people  yawning  and  nodding  off,  and  not  always 
participating  and  that  always  frustrated  me;  because  I thought  'why  are  they  sitting  here  and  not  doing  something?' 

It  bothered  her  when  she  found  that  students  were  not  actively  involved  in  the  learning  process.  A socio- 
cognitive  cons  true  tivi  Stic  approach  to  learning  emphasizes  that  learning  is  an  active  and  evolving  process.  Learners 
are  constantly  engaged  in  integrating  new  information  into  existing  knowledge  structures.  Through  the  on-going 
interaction  between  teacher  and  student,  development  of  meaningful,  valid  and  increasingly  complex  knowledge 
structures  are  encouraged.  This  demands  two-way  communication  where  students  attempt  to  explain  their 
interpretation  and  listen  to  others’  understanding.  The  teacher  in  the  study  believed  that  it  was  important  for  her  to 
get  students  involved  in  the  course,  and  actively  engaged  in  the  learning  process.  She  also  wanted  flexibility  in  what 
the  students  learned  as  well  as  how  they  learned.  This  fits  in  with  the  Deweyan  notion  that  children  learn  both  inside 
and  outside  of  school  (Dewey,  1938).  In  this  situation,  students  are  literally  learning  outside  school.  Being  physically 
outside  the  school,  they  are  able  to  bring  their  outside  experiences  into  the  classroom. 

While  the  teacher  used  flexible  learning  strategies,  there  is  a paradoxical  need  for  her  to  maintain  control  of 
the  classroom.  From  her  personal  experience,  this  teacher  believed  that  an  organized  learning  environment  was  not 
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necessary  for  students  to  learn  because  students  would  learn  when  they  found  the  need  to  learn  something  that  is  of 
interest  to  them.  Yet,  at  the  same  time,  the  teacher  had  a personality  that  wanted  to  be  in  control  of  the  learning 
environment.  The  teacher  provides  reasoning  for  this  by  providing  information  about  her  own  educational 
background.  The  following  is  another  excerpt  from  her  interview: 

Teacher.  We  moved  every  year.  I went  to  13  or  14  different  schools  and  one  year  we  moved  in  May  and  I went  to 
another  city  and  another  school  and  1 wasn't  doing  the  same  work  they  had  done  and  so  they  put  me  back,  so  I had 
to  do  fifth  grade  again.  1 was  so  humiliated.  So  I repeated  fifth  grade.  Then  we  went  to  Switzerland.  In  sixth  grade  I 
did  about  2 weeks  of  school.  My  mother  was  supposed  to  home  teach  me,  but  she  didn’t  manage  to,  and  so  that 
was  a whole  loss  of  a year  and  at  the  end  of  that  year,  we  moved  back  to  NY  and  I had  2 weeks  of  school  and 
again,  it  was  the  end  of  this  year,  and  I was  in  6,h  grade.  They  just  went  ahead  and  put  me  in  7lh  grade  the  next  year 
and  I was  there  for  2 weeks  and  they  put  me  in  8,h  grade.  So  1 basically  repeated  5th  grade  twice  and  the  next  class  I 
was  in  was  the  8th  grade.  So  I had  a very  unconventional  education  and  basically  felt  that  whatever  I needed  to 
know,  I learned,  whenever  I wanted  to  learn  it.  So  1 felt  that  organized  education  was  a big  waste  of  time.  And  I am 
sure  that  is  what  underlies  my  whole  philosophy  and  why  1 like  this  medium.  Although  it  is  organized,  there  is 
flexibility  for  the  students  to  work  through  it... Again,  the  distance  ed.  courses  that  are  not  successful,  are  the  ones 
that  are  not  very  well  structured. 

The  teacher  borders  on  extreme  child-centered  learning  here.  She  relates  her  personal  experiences  and 
believes  that  she  learned  a lot  even  without  a so-called  structured  education.  She  however  qualifies  her  experience 
and  relates  it  to  the  new  technological  environment  that  she  teaches  in.  She  is  quick  to  point  out  that  structure  is  an 
important  component  of  the  distance  education  metaphor.  She  therefore  believes  that  having  structure  is  important  in 
a course,  again  consistent  with  the  holistic  paradigm.  Structure,  in  this  context  is  different  from  control.  Control  is 
determined  by  a single  authority  figure,  whereas  structure  can  be  emergent.  This  teacher  provides  flexibility  inside 
an  organized  environment  and  underlies  the  fact  that  structure  is  different  from  chaos.  For  example,  she  allowed 
students  to  select  questions  for  discussion,  given  a general  topic  area  to  work  with.  Thus,  some  of  the  content  of 
learning  is  also  emergent.  Dewey  believed  that  it  is  miseducational  for  teachers  to  impose  content  on  the  learner 
(Dewey,  1938).  The  teacher  also  realized  this.  She  wanted  students  to  see  the  relevance  of  what  they  were  learning. 
Yet,  she  reserved  final  judgement  on  what  they  learned.  This  is  again  an  important  indicator  that  the  teacher’s 
position  with  regard  to  content  was  close  to  critical  democracy  (Goodman,  1992). 

In  warranting  the  above  assertion,  the  researchers  examined  the  data  and  found  that  this  teacher  believes  in 
providing  the  students  with  the  ability  to  learn  flexibly.  Her  epistemological  position  may  be  identified  as 
‘phenomenological’.  She  had  structured  her  course  in  the  transaction  position  (Miller  and  Seller,  1985).  In  the 
transaction  position,  the  student  is  seen  as  rational  and  capable  of  intelligent  problem  solving.  Education  is  viewed  as 
a dialog  between  the  student  and  the  curriculum  in  which  the  student  reconstructs  knowledge  through  a dialogical 
process.  The  historical  antecedent  is  the  period  of  the  Enlightenment,  and  the  influential  people  most  closely  allied 
with  the  transaction  position,  according  to  Miller  and  Seller,  are  Horace  Mann,  John  Dewey,  and  Jean  Piaget.  The 
teacher  provided  opportunities  for  students  to  collaborate  with  each  other  and  structured  appropriate  learning 
experiences  in  the  form  of  group  discussions  conducted  through  the  conferencing  system.  Further,  the  teacher 
encourages  (on  certain  weeks  even  requires)  students  to  discuss  some  aspect  of  the  learning  content  among 
themselves. 

There  is  a strong  emphasis  on  giving  the  students  an  opportunity  to  interpret  what  they  learn.  Meaning- 
making is  again  central  to  the  holistic  tradition.  Dewey  believed  that  the  role  of  the  teacher  is  to  ‘guide’  the 
educational  process,  not  ‘teach’  it.  This  teacher  encouraged  students  to  explore  topics  within  the  general  area  of 
learning.  Although  she  gave  students  the  opportunity  to  choose  what  they  wanted  to  learn  (negotiation),  the 
negotiation  was  allowed  within  the  framework  of  content  and  structure  provided  by  the  teacher.  Yet,  this  teacher 
thought  that  her  personality  was  oriented  towards  being  more  of  a controlling  person.  She  mentioned  that  although 
she  likes  to  take  control  in  a traditional  class,  she  made  a special  effort  not  to  do  that  in  the  online  class.  Therefore, 
she  achieved  a delicate  balance  between  control  she  exerted  on  the  curriculum,  and  flexibility  that  she  allowed 
students  in  the  online  medium. 

The  learning  experience  provided  in  an  online  environment  is  therefore  determined  by  the  type  of  activities 
encouraged  in  the  class.  The  teacher  determines  these  activities  and  other  aspects  of  course  structure.  The  teacher's 
personality  and  background  have  a bearing  on  her  ideas  of  what  learning  is  and  her  epistemological  position. 
Consequently,  this  has  influence  on  the  way  the  teacher  designs  her  course  and  the  activities  that  she  includes  in  the 
online  learning  environment.  Again,  course  structure  also  has  direct  bearing  on  the  role  the  teacher  takes  on  in  an 
online  class  and  the  time  she  spends  on  it. 

Conclusions 

The  lack  of  communication  between  student  and  teacher  in  the  distance  education  environment  is  one  of  the 
most  debated  topics  in  the  field.  Although  the  characteristics  of  each  technology  affect  the  ability  of  participants  to 
communicate,  the  choice  of  the  technology  to  be  used  in  the  class  was  a personal  decision  made  by  the  teacher  of  the 
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course.  While  a conferencing  system  provides  a means  of  communication,  simply  having  the  means  of 
communication  does  not  mean  that  effective  communication  takes  place.  One  needs  to  assess  the  type  and  quality  of 
interaction  that  does  take  place  using  these  media.  The  course  that  I observed  uses  computer  conferencing  as  a 
medium  of  delivery  to  ensure  that  students  have  an  opportunity  to  interact  with  the  instructor.  The  success  of  such  a 
system  would  depend  on  the  quality  of  the  feedback  that  the  teacher  provides,  and  learning  environment  the  teacher 
creates  for  the  students.  The  implementation  of  such  a system  does  not  guarantee  a better  quality  of  learning,  it  at 
least  opens  up  an  avenue  of  communication  between  teachers  and  students.  This  again  underscores  the  idea  that 
philosophy  and  personality  are  definite  factors  that  determine  how  teachers  structure  their  courses.  The  teacher's 
beliefs  guided  her  in  the  process  of  course  design;  therefore,  her  teaching  philosophy  affected  the  way  she  has 
designed  the  course. 

A recent  naturalistic  research  study  (Annand,  1998)  that  examined  six  instructors'  experiences  with 
computer  conferencing,  documented  how  these  instructors  understood  the  educative  process  in  computer 
conferencing  and  its  relationship  to  themselves  and  their  students.  The  findings  reveal  that  instructors'  varied 
personal  philosophies  of  learning  were  foundational  in  delineating  their  relationship  with  the  technology.  But  other 
aspects,  such  as  recognition  of  students'  learning  styles,  the  instructors'  repertoire  of  preferred  pedagogical  strategies, 
and  the  discourse  patterns  privileged  by  computer  mediated  communication  (CMC),  were  also  important  influences 
in  their  understanding  of  students'  computer  conferencing  practices.  In  the  present  study,  the  teacher’s  philosophy, 
personality,  and  technology  create  a complex  interplay  of  factors  that  influence  how  the  teacher  structures  her 
course.  Her  beliefs  guided  her  in  the  process  of  course  design;  therefore,  her  teaching  philosophy  affected  the  way 
she  designed  the  course.  The  technology  she  was  using  to  teach  the  online  class  also  had  an  undeniable  effect  on  how 
she  taught  her  distance  class. 

Teachers  who  teach  the  online  course  perceive  the  need  to  do  a lot  of  additional  work  for  such  a course. 
While  they  could  have  spent  less  time  teaching  the  online  course,  they  felt  that  the  amount  of  time  spent  contributed 
to  a worthwhile  teaching  experience.  This  was  mainly  because  the  online  metaphor  gives  students  and  teachers  great 
flexibility  in  terms  of  participation.  Also,  students  were  more  satisfied  with  the  course  and  participated  actively  in 
the  conferencing  system  because  they  felt  that  the  teacher  was  more  ’present*  and  available  for  them,  and 
consequently,  the  teacher  felt  that  she  was  able  to  get  more  out  of  the  students. 

In  conclusion,  structure  and  dialog  are  inherently  related,  as  pointed  out  by  Moore  (1991).  Further,  teacher 
personality,  emotions  and  philosophy  are  factors  that  affect  course  structure  in  an  online  course.  Dialog,  of  course, 
relies  on  the  ability  of  the  students  to  interact  with  each  other  and  the  teacher,  but  dialog  needs  to  be  fostered  and 
nurtured  by  the  teacher.  Interaction  has  to  occupy  an  important  place  in  the  design  of  the  course  and  only  the  teacher 
can  give  it  such  a status  by  making  it  a priority.  All  decisions  she  makes  with  regard  to  the  structure  of  the  course 
have  a direct  impact  on  dialog.  It  would  be  interesting  to  examine  the  degree  to  which  teacher  personality  is  a factor 
that  determines  course  success  regardless  of  the  limitations  and  structure  imposed  by  technology  and  other  factors. 
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Effects  of  Faculty  Motivation  dm  Distributive  Education  Environments  at  Institutions 

of  Higher  Education 


Marlon  R.  Mitchell 
Indiana  University 

As  the  distributive  education  environment  (DE)  grows  and  develops  many  changes  in  current  administrative 
policy  and  faculty  approaches  to  instruction  will  need  to  evolve  to  accommodate  this  method  of  delivery. 

Distributive  education  is  a form  of  instruction  that  is  dynamic,  using  many  modes  of  delivery.  This  delivery  may  be 
synchronous  or  asynchronous  and  takes  into  account  alternative  methods  available  to  the  faculty  (internet,  lecture, 
video,  audio,  etc.).  However,  utilizing  these  alternative  methods  usually  requires  a more  demanding  workload  for  the 
faculty.  The  heavier  workload  is  a result  of  designing,  developing,  implementing,  and  evaluating  the  materials 
delivered  via  DE. 

This  research  will  attempt  to  evaluate  three  factors  that  may  underlie  faculty  resistance  to  distributive 
education: 

1 . Difference  in  motivation  in  faculty  according  to  career  status  (junior,  senior,  departmenthead), 

2.  Difference  in  Administrative  and  faculty  perceptions  of  reward  structure,  anu 

3.  Difference  in  motivation  to  teach  via  distance  (intrinsic  and  extrinsic). 

Research  suggests  that  these  themes  are  recurring  in  institutions  of  Higher  Education  (Blackburn  & 
Lawrence,  1995).  Blackburn’s  findings  neither  support  nor  reject  themes  one  and  two.  Blackburn  suggests  that 
faculty  call  on  one  or  the  other  reward  system  depending  on  factors  and  circumstances  neither  theory  alone 
adequately  takes  into  account.  BlackbunTalso  suggests  that  socialization  is  a recurring,  if  not  continuous, 
phenomenon.  Faculty  socialization  begins  in  graduate  school  and  continues  throughout  their  career.  Senior  faculty, 
administration,  and  other  cultural  factors  of  the  institution  indoctrinate  faculty.  If  the  faculty  changes  institutions  re- 
socialization often  happen. 

The  resources  necessary  for  successful  development  and  implementation  of  DE  are  not  the  focus  of  this 
research.  This  research  examines  the  chief  factor  underlying  effective  instruction,  the  faculty.  A primary  barrier 
determining  the  future  of  DE  may  be  the  faculty’s  attitudes,  perceptions,  and  willingness  to  use  technology.  The 
intention  therefore  is  to  determine  if  a link  exists  between  motivation  and  the  use  of  technology.  There  appears  to  be 
a need  to  increase  the  intrinsic  motivations  of  faculty  to  use  the  technology;  therefore,  minimizing  the  cost  of 
extrinsic  rewards  (Deci,  Eghrari,  Patrick,  & Leone,  1994).  Increasing  intrinsic  motivation  and  decreasing  extrinsic 
rewards  may  lead  to  higher  perceived  self-efficacy. 

“Judgements  of  one’s  capabilities  can  affect  rate  of  skill  acquisition,  and  performance  mastery,  in  turn,  can  boost 

self-efficacy  in  a mutually  enhancing  process”,  (Bandura  & Schunk,  1981). 

Cultivating  faculty  self-efficacy  may  produce  higher  quality  instruction  and  increase  faculty  and  institutional 
efficacy  (Newman  & Tuckman,  1997).  If  this  has  some  truth,  then  efforts  in  developing  faculty  motivation  must 
emphasize  the  importance  of  faculty  and  institutional  efficacy.  Olcott's  research  supports  this  assertion. 

Don  Olcott’s  Institutional  Support  Framework  (Olcott  & Wright,  1995)  examines  an  institutional  faculty 
support  system.  The  framework  consists  of  structures  ranging  from  evaluation,  modes  of  delivery  and  support, 
administrative  duties,  reward  systems,  and  faculty.  The  framework  has  five  levels  represented  by  rings. 

In  this  article,  the  focus  is  on  the  three  innermost  rings  of  Olcott’s  framework.  The  commonality  that  binds 


this  structure  is  their  direct  effect  on  faculty  motivation.  The  core  of  the  structure  represents  the  faculty. 
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The  next  ring  represents  promotion  and  tenure,  compensation,  training,  and  release  time.  The  rings 
described  above  can  be  utilized  to  help  the  faculty  internalize  the  extrinsic  rewards  each  represents.  The 
internalization  can  have  positive  effects  on  self-efficacy. 

The  administrative  bodies  are  best  able  to  deal  with  institutional  efficacy.  The  outermost  ring  of  our  focus 
represents  faculty  senate,  deans,  president/provost,  and  chairs.  The  outermost  rings  are  bodies  that  may  effect  faculty 
motivation.  Once  the  institution  believes  that  faculty  motivation  is  of  critical  importance  to  its  growth  and 
development,  then  the  change  permeates  the  institution  in  a top  down  approach.  This  change  will  be  reflected  in  the 
second  ring  of  Olcott's  framework. 

“Despite  the  recent  expansion  of  distance  education  programs,  many  faculty  continue  to  resist  participation  in 
distance  teaching.  We  assert  that  this  resistance  has  been  due,  in  large  part,  to  the  lack  of  an  institutional  support 
framework  to  train,  compensate,  and  reward  distance  teaching  faculty  at  levels  commensurate  with  those  in 
traditional  instructional  roles"(01cott  & Wright,  1995). 

To  maximize  faculty  participation  their  personal  motivation  must  be  increased  to  a level  of  satisfaction.  To 
minimize  the  cost  of  externalities  (money,  time,  etc.)  the  reward  structure  must  lend  itself  to  cultivating  intrinsic 
motivators  and  internalizing  external  motivators  (Bandura,  1991).  A person  is  intrinsically  motivated  when  no 
apparent  rewards  are  received  except  for  the  activity  itself.  Intrinsic  motivation  might  be  either  innate  or  learned. 
Education  is  generally  said  to  be  an  intrinsically  motivated  profession.  Many  primary  and  secondary  educators  have 
been  known  to  work  in  precarious  environments,  educating  children  despite  obstacles  and  adversity.  For  them, 
teaching  in  such  conditions  is  worth  it  even  if  only  one  child  leaves  more  knowledgeable  then  when  they  began. 

Their  gratification  does  not  come  in  the  form  of  salary  or  benefits;  their  reward  comes  from  the  act  of  teaching.  On 
the  other  end  of  the  continuum  we  have  external  rewards.  External  rewards  tend  to  reduce  intrinsic  motivation  (Deci, 
1971).  In  addition,  the  administrators  must  take  on  a significant  role  in  supporting  the  reward  structure  with  policy 
and  finance. 

Faculty  motivation  is  a topic  that  has  re-emerged  out  of  an  attempt  to  restructure  the  institution  of  education 
to  fit  the  non-traditional  method  of  instructional  delivery  commonly  known  as  distance  education  (Clark,  1993). 
Faculty  have  always  been  considered  an  invaluable  resource,  but  often  our  most  precious  resources  are  neglected  and 
taken  for  granted  (Dillon  & Walsh,  1992).  Findings  from  a study  by  Taylor  & White  (1991)  suggest  that  faculty  are 
generally  motivated  by  intrinsic  rewards  rather  than  extrinsic  rewards  when  evaluating  the  benefits  of  distance 
education.  They  studied  an  Australian  higher  education  institution,  where  faculty  is  required  to  work  in  the 
conventional  face  to  face  setting  as  well  as  to  prepare  materials  for  use  by  distance  education.  The  findings  suggest 
that  faculty  derived  job  satisfaction  from  teaching  via  distance  education.  The  following  five  ratings  were  consistent 
in  importance  in  terms  of  faculty  attitudes  toward  teaching  in  the  distance  education  mode:  quality  of  interaction  with 
students;  working  with  motivated  students;  satisfaction  from  the  act  of  teaching;  feeling  of  personal  achievement;  and 
high  level  of  student  outcomes  (Taylor  & White,  1991). 

Attitudes  of  American  faculty  in  higher  education  institutions  are  consistent  with  Australian  faculty. 

In  a study  of  telecourse  faculty  at  an  American  university, 

“A  few  faculty  will  continue  to  utilize  telecommunications  to  teach  students  at  a distance  for  a variety  of  personal 
reasons,  ranging  from  diversity  of  experience  to  an  altruism  toward  the  nontraditional  learner”  (Dillon,  1989). 

This  quote  supports  the  view  that  a few  faculty  members  are  intrinsically  motivated  to  teach,  whether  an 
external  reward  is  provided  or  not.  However,  this  does  not  mean  that  teaching  does  not  require  external  rewards. 

In  a survey  of  317  faculty,  thirty  seven  percent  of  faculty  expressed  a belief  that  they  would  not  be 
adequately  rewarded  for  their  efforts  in  distance  education  (Clark,  1993).  A national  study  of  4,000  full-time,  tenure 
track  faculty  generally  agreed  that  research  is  heavily  rewarded  in  merit  pay  increases,  tenure,  and  promotion 
decisions  (Fairweather,  1993).  It  also  shows  faculty  believed  that  adequate  teaching  is  a necessary  but  not  sufficient 
condition  for  tenure.  Therefore,  faculty  has  a motive  to  avoid  devoting  extensive  amounts  of  time  to  teaching 
(distance  education,  lecture,  etc.)  (Kasten,  1984). 

Barriers  (lack  of  time,  increase  in  workload,  technical  problems)  to  distance  education  may  contribute  to  the 
decline  of  faculty  motivation  (Dillon  & Walsh,  1992).  Faculty  may  create  their  own  barriers  to  using  distance 
education;  therefore,  directing  their  efforts  to  other  aspects  of  academia: 

“It  also  appears  that  faculty  career  stages  do  not  allow  for  any  adoption  to  occur,  either.  For  faculty  in  the  early 
entry  and  career  stabilization  stage,  the  importance  of  obtaining  tenure  dramatically  outweighs  the  experimentation 
with  a questionable  and  little  researched  teaching  innovation.  The  midlife  stage  faculty  does  not  accept  the  thought 
of  style  of  teaching  in  front  of  a classroom  that  has  taken  several  years  to  develop  and  has  rewarded  them  with  the 
necessary  tenure  and  promotions.  The  senior  life  stage  faculty  is  no  different.  The  senior  professor  realizes  that  a 
high  merit  increase  closer  to  retirement  is  more  important  than  a change  of  a teaching  style”  (Koontz,  1989). 

It  appears  that  in  order  to  get  faculty  to  incorporate  DE  into  their  method  of  instruction  their  motives  must 
change.  This  leads  us  to  explore  research  in  the  field  of  motivation. 
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To  begin  we  will  examine  the  research  surrounding  intrinsic  motivation  and  extrinsic  rewards.  Intrinsic 
motivation  and  extrinsic  rewards  are  extremely  important  in  the  struggle  to  overcome  the  barriers  to  DE.  In  order  to 
facilitate  the  growth  and  development  of  DE  a balance  between  the  two  is  needed.  The  research  has  shown  that 
offering  extrinsic  rewards  may  diminish  intrinsic  motivation  (Deci,  1971).  However,  research  has  also  shown  that 
extrinsic  rewards  may  also  increase  intrinsic  motivation  when  linked  to  goals  (Heyman  & Dweck,  1992).  Goals 
setting  theory  states  that  common  human  behavior  is  purposeful;  therefore,  regulated  by  an  individuals  (Latham  & 
Locke,  1991).  Goal  setting  has  at  least  three  attributes  of  motivated  action:  (1)  direction:  goals  are  directed  by 
relevancy  of  action,  (2)  intensity:  effort  is  regulated  or  expended  accordingly  to  difficulty  of  goal,  and  (3)  duration: 
goals  are  effected  by  time  (Latham  & Locke,  1991).  Goal  setting  and  motivation  research  shows  that  proximal 
(immediate)  goals  can  develop  self-efficacy  and  intrinsic  motivation  (Bandura  & Schunk,  1981).  It  has  also  been 
shown  that  personally  setting  distal  (distant)  goals  can  have  a positive  effect  on  intrinsic  motivation  (Manderlink  & 
Harackiewicz,  1984).  Distal  goals  can  increase  institutional  efficacy  if  the  leaders  provide  them  to  their  employee’s 
and  use  proximal  goals  as  benchmarks  (Latham  & Locke,  1991).  Participant  modeling  with  verbal  persuasion  can 
promote  self-efficacy  and  task  performance  (Newman  & Tuckman,  1997). 

Methodology 

This  research  utilizes  a survey  to  gather  data  about  faculty  motivation  towards  distance  education.  Indiana 
Higher  Education  Telecommunications  System  was  chosen  as  the  group  to  sample.  Because  this  group  reflected 
characteristics  of  the  target  population  of  the  study  which  included;  l)people  in  higher  education,  2)that  were 
involved  in  distributive  education  environments,  3)and  were  in  both  administrative  and  faculty  positions.  The  sample 
for  the  study  was  selected  from  a list  of  statewide  technology  users  (Faculty  and  Administrators)  who  subscribe  to 
Indiana  Higher  Education  Telecommunications  Services  (IHETS)  for  delivery  of  instructional  services.  IHETS 
services  include  internet,  video  down  link,  and  audio-graphics.  IHETS  maintains  a distribution  list  of  administrators 
and  faculty  that  use  their  services. 

The  questionnaire  was  distributed  via  E-mail  and  included  both  the  closed-form  which  permits  limited 
responses  (multiple-choice  questions)  and  open-form.  The  open  form  allows  any  comments  that  the  participant  may 
want  to  contribute.  The  survey  methodology  was  chosen  because  of  its  usefulness  in  data  collection.  It  allowed  for 
the  distribution  of  the  same  instrument  (questionnaire)  to  all  of  the  participants  of  the  study.  The  questionnaire  also 
aided  in  the  data  analysis  strategy  chosen.  A descriptive  research  strategy  was  employed  to  summarize  the  data 
collected.  The  data  collection  instruments  were  delivered  via  electronic  mail  to  an  IHETS  distribution  list  of  faculty 
and  administrators.  Approximately  two  hundred  people  subscribe  to  this  distribution  list.  At  the  time  of  the  survey, 
the  administrator  of  the  listserve  confirmed  that  only  fifty  to  seventy-five  of  the  two  hundred  subscribers  were  active 
participants.  Thirty-two  people  responded  to  the  survey,  of  the  respondents  thirteen  were  usable.  Surveys  that  were 
incomplete  were  not  used.  Faculty  and  administrators  from  eight  higher  education  institutions  in  the  state  of  Indiana 
actively  contribute  to  the  listserv.  This  listserv  is  a place  that  subscribers  can  discuss  issues  surrounding  distance 
education.  An  invitation  to  participate  (Appendix  A)  in  the  study  was  sent  to  the  distribution  list  followed  by  a 
survey  questionnaire  (Appendix  B). 

The  survey  was  created  using  a couple  of  strategies.  First,  a collection  of  previous  surveys  and 
questionnaires  were  analyzed  and  then  synthesized  into  an  instrument  that  was  then  pre-tested  with  two  faculty 
members.  Secondly,  the  results  of  the  pre-test  were  taken  into  consideration  and  the  assistance  of  a campus  research 
statistical  service  was  employed.  The  result  of  which  was  distributed  to  the  participants  of  this  research. 

The  collection  device  (questionnaire)  centered  around  three  themes:  faculty  demographics,  job  satisfaction, 
and  early  socialization.  Faculty  and  administrators  were  asked  to  respond  to  the  questions  and  return  them  in  a 
relatively  short  period  of  time,  approximately  one  week.  During  that  time  frame,  reminders  were  sent  to  expedite  the 
return. 

Quantitative  methods  of  data  analysis  were  employed  to  synthesize  results.  Statistical  Package  for  Social 
Science  (SPSS)  was  used  to  collect  and  analyze  the  data.  The  following  statistical  functions  were  used  to  give  an 
accurate  description  of  the  data  collected  and  analyzed  frequency  distribution,  factor  analysis,  and  t-Test.  Three 
statistical  procedures  were  used  frequency  distribution,  factor  analysis,  and  t-Test s respectively.  Frequency 
distribution  more  commonly  known  as  frequency  count  recording  is  defined  as  the  measurement  of  the  number  of 
times  that  each  observation  variable  occurs  during  an  event  (Borg,  Gall,  Borg  page  760).  Secondly,  in  an  effort  to 
narrow  the  scope  of  variables  a statistical  procedure  called  factor  analysis  was  applied.  Factor  analysis  is  a statistical 
procedure  for  reducing  a set  of  measured  variables  to  a smaller  number  of  variables  (called  factors  or  latent 
variables)  by  combining  variables  that  are  moderately  or  highly  correlated  with  each  other  (Gall,  Borg,  Gall  page 
759).  Third,  a t-Test  is  a test  of  statistical  significance  is  used  to  determine  whether  the  null  hypothesis  that  two 
sample  means  come  from  identical  populations  can  be  rejected  (Gall,  Borg,  Gall  page  772).  Statistical  significance  is 
evaluated  by  the  power  that  it  holds.  The  term  power  refers  to  the  ability  of  a statistical  analysis  to  reduce  Type  II 
errors.  Higher  power  tests  are  more  likely  to  find  statistical  relationships  when  there  is  one  (Vockell  & Asher,  1995). 
We  will  discuss  our  views  and  discoveries  resulting  the  methodology  employed. 
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Discussion  and  Analysis 

Faculty  motivation  is  not  a new  topic;  however,  its  is  an  old  one  that  has  been  rejuvenated  in  light  of  distributive 
education.  In  order  for  DE  to  take  a stronghold  in  education  the  primary  resource  (faculty)  must  adopt  it  and 
implement  it  into  their  pedagogy.  This  change  creates  a need  to  find  the  source  of  faculty  motivation.  Our  restated 
question  is  “What  effect  does  DE  have  on  faculty  motivation”? 

In  answering  the  above  question  an  evaluation  of  three  factors  was  conducted: 

1 . Difference  in  motivation  of  faculty  according  to  career  status  (junior,  senior,  departmenthead), 

2.  Difference  in  Administrative  and  faculty  perceptions  of  reward  structure,  and 

3.  Difference  in  motivation  to  teach  via  distance  (intrinsic  and  extrinsic). 

Below  are  distributions  that  are  supportive  of  the  evaluation  factors: 

Table  1 Rank 


RANK 

Frequency 

Percent 

Valid 

Percent 

Cumulative 

Percent 

Valid  Non-tenured 

3 

23.1 

25.0 

25.0 

Tenured 

8 

61.5 

66.7 

91.7 

Administrative 

1 

7.7 

8.3 

100.0 

Total 

12 

92.3 

100.0 

Missing  System 

Missing 

1 

7.7 

Total 

1 

7.7 

Total 

13 

100.0 

Table  1 shows  that  23.1%  of  the  respondents  were  non-tenured  or  junior  faculty,  61.5%  were  tenured,  and 
7.7%  were  administrators. 

Table  2 Motivation 


Frequenc 

y 

Percent 

Valid 

Percent 

Cumulative 

Percent 

Valid  Intrinsic 

8 

61.5 

66.7 

66.7 

Extrinsic 

4 

30.8 

33.3 

100.0 

Total 

12 

92.3 

100.0 

Missing  System 

Missing 

1 

7.7 

Total 

1 

7.7 

Total 

12 

100.0 

Table  2 gives  us  a little  more  insight  in  to  what  their  motivations  are.  Our  findings  are  61.5%  of  the 
respondents  say  that  their  source  of  motivation  is  personal  fulfillment.  30.8%  answered  that  an  opportunity  to  help 
student  learn  via  DE  is  their  gratification.  A total  of  92.3%  of  the  respondents  provided  answers  that  are  in  line  with 
our  definition  of  intrinsic  motivation.  It  seems  highly  likely  that  the  majority  of  faculty  are  intrinsically  motivated 
regardless  of  rank. 

In  an  effort  to  determine  what  faculty  desired  (money,  time,  assistance,  or  formal  recognition)  the  most 
when  creating  a DE  69.2%  responded  that  time  is  most  desired,  followed  by  assistance,  money,  and  recognition 
respectively.  This  supports  the  idea  that  faculty  are  intrinsically  motivated.  Faculty  were  also  asked,  “If  you  were  to 
leave  your  job  to  accept  another  position,  would  you  want  to  do  more,  less,  or  about  the  same  teaching  via 
distance?”,  92.3%  responded  that  they  would  like  do  about  the  same  or  more  (see  Table  3). 

Table  3 Amount 


Frequenc 

y 

Percent 

Valid 

Percent 

Cumulative 

Percent 

Valid  More 

6 

46.2 

50.0 

50.0 

Same 

6 

46.2 

50.0 

100.0 

Less 

0 

0 

0 

Total 

12 

92.3 

100.0 

Missing  System 

Missing 

1 

7.7 

Total 

1 

7.7 

Total 

13 

100.0 

Faculty  was  also  asked  what  were  the  major  support  issues  that  they  have  encountered  at  their  institution. 
The  majority  responded  that  technical  support  was  the  most  encountered. 

The  above  frequency  distribution  analysis  gives  a foundation  upon  which  we  can  continue  to  build  our  case 
about  faculty  motivation. 
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Factor  analysis  was  used  to  determine  where  improvements  could  be  made  to  the  survey  questionnaire. 
Changes  were  made  according  to  the  clusters  that  were  generated  using  the  Principal  component  analysis  extraction 
method  with  varimax  normalization.  Due  to  the  time  constraint  of  the  study,  the  modified  survey  could  not  be 
redistributed  to  the  sample. 

Table  4 


Rotated  Component  Matrix  ■ 


Component 

1 

2 

3 

.761 

.125 

.234 

SFSUP 

.747 

-.192 

-6.65E-02 

FASUP 

.713 

.291 

-.297 

STSUP 

.703 

.169 

.133 

TEREWARD 

-.667 

6.314E-03 

.188 

REREWARD 

-.615 

-.356 

-.389 

EXASSIST 

-.564 

.361 

.543 

ADREWARD 

.114 

.856 

9.282E-02 

DEREWARD 

-.415 

.779 

-9.24E-02 

TESUP 

.233 

.677 

.143 

EXMONEY 

-.362 

-.628 

.533 

SEREWARD 

.303 

.138 

.844 

EXRECOG 

-.165 

-.164 

.832 

MOTIVE 

-8.03E-03 

-5.57E-02 

-.489 

Extraction  Method:  Principal  Component  Analysis. 
Rotation  Method:  Varimax  with  Kaiser  Normalization. 


a-  Rotation  converged  in  5 iterations. 

The  Time  and  Support  Factor  shows  that  faculty  value  support  and  time  in  their  creation  of  DE,  we  can  see 
this  factor  by  analyzing  the  loads  associated  with  the  first  six  variables.  The  loads  are  as  follows;  .761  time  (value 
time  to  create  DE),  .747  support  (staff  support  resources),  .713  faculty  support  (major  support  issue),  .703  student 
support  (student  support  resources),  -.667  teaching  rewards  (institutional  reward),  -.615  research  (institution  reward). 
This  information  tells  us  the  difference  of  perception  between  faculty  and  the  administration.  Faculty  value  and  place 
importance  on  time  and  support  to  create  and  deliver  DE,  but  the  institution  rewards  research  and  traditional  teaching 
methods. 

The  External  Reward  factor  provides  us  with  additional  information  that  shows  us  the  gap  that  exists 
between  faculty  and  administration.  The  loads  are  as  follows;  .856  administrative  reward  (institution  rewards),  .779 
distance  education  reward  (institution  rewards),  .677  technical  support  (major  support  issue),  and  -.628  money  (value 
most  in  creating  a DE  course).  Money  has  a negative  load  value,  this  may  be  interpreted  to  mean  that  in  regards  to 
other  factors  the  reward  of  money  does  not  outweigh  the  reward  of  factors  (time,  assistance,  and  recognition)  that 
support  the  development  of  internal  motivation. 

The  loads  associated  with  the  Service  and  Recognition  factor  are;  .844  service  reward  (institution  rewards) 
and  .832  recognition  (value  most  in  creating  a DE  course).  These  loads  show  that  faculty  also  value  performing 
service  and  being  recognized  for  their  efforts  in  DE. 

Our  analysis  of  the  factors  paints  a picture  that  is  quite  easy  to  interpret.  The  reality  of  the  problem  is  that 
faculty  and  administration  have  different  perceptions  of  what  is  important  to  each  other.  The  faculty  values  time  and 
assistance  when  working  in  DE.  On  the  other  extreme  administrators  value  things  that  are  institutionally  rewarded, 
such  as  research  and  how  many  students  are  in  the  classroom.  However,  we  must  not  stop  here  further  analysis  need 
to  be  done  to  sharpen  our  focus. 

Third,  a t-Test  was  applied  to  the  data  to  test  for  significance.  Although,  t-Tests  were  conducted  no 
significance  was  found.  This  can  be  attributed  to  lack  of  power  due  to  a small  data  pool.  However,  this  does  not 
mean  that  the  study  is  not  significant.  It  does  mean  that  further  research  with  a larger  data  pool  must  be  conducted  in 
order  to  verify  significance.  The  results  from  the  frequency  distribution  and  factor  analysis  tests  lend  credence  to  a 
strong  possibility  of  significance.  Currently,  we  can  only  suggest  that  with  a larger  data  pool  statistical  significance 
would  be  the  result. 

It  appears  that  faculty  are  intrinsically  motivated  to  teach  in  DE.  This  finding  should  not  determine  the  end 
of  Administrator  and  Policymaker  efforts  to  create  a fair  and  equitable  environment  for  faculty.  Administrators  and 
Policymakers  should  take  this  opportunity  to  create  an  atmosphere  that  fosters  intrinsic  motivation.  This  does  not 
imply  that  faculty  are  not  concerned  with  promotion  and  tenure  or  other  external  rewards.  The  reality  is  that  external 
rewards  are  extremely  important  to  their  existence,  but  personal  fulfillment  is  equally  as  important.  Therefore, 
Administrators  and  Policymakers  must  work  with  faculty  to  reach  a balance  between  external  rewards  and  intrinsic 
motivators  (Olcott  & Wright,  1995). 


Motivation  theory  supplies  a foundation  upon  which  faculty  socialization  to  DE  can  be  built.  Motivation 
offers  many  theories  that  may  help  build  institutional  and  self-efficacy;  thereby,  increasing  intrinsic  motivation  and 
internalizing  external  motivators.  Theories  such  as  goal  setting,  participant  modeling,  and  intrinsic  motivation  can  all 
be  used  to  foster  an  efficacious  environment. 
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Abstract 


The  development  of  effective  web-based  learning  environments  requires  careful  assessment  of  the  impact  of 
design  variables  on  student  learning.  The  present  study  investigates  the  effects  and  interactions  of  web- 
page design  on  student  navigation  and  decision  making  in  web-based  learning  environments.  The 
following  research  questions  are  examined  via  quantitative , qualitative,  and  exploratory  data  analysis 
procedures: 

* Does  instructional  grouping  or  web  page  format  effect  navigational  decisions  in  an  online  learning  environment? 

* Are  there  differences  in  the  number  of  unique  URLs  visited  with  respect  to  instructional  grouping  or  web  page  format? 

* Are  there  differences  in  student  attitude  toward  instructional  grouping  or  computer-mediated  instruction? 

* Are  there  differences  in  quality  of  students  written  language  with  respect  to  instructional  grouping  or  web  page  format? 

Fifty  undergraduate  students  enrolled  in  an  introductory  educational  survey  class  at  a large  southwestern 
community  college  served  as  subjects.  Each  subject  was  randomly  assigned  to  one  of  four  groups; 
cooperative  learning  with  advisement,  cooperative  learning  without  advisement,  individual  learning  with 
advisement,  and  individual  learning  without  advisement.  Subjects  participated  in  computer  mediated 
instruction  employing  Internet  resources  (World  Wide  Web)  to  gather  information  and  compose  an  essay. 
Analysis  of  Variance  and  Chi  Square  procedures  were  applied  to  assess  the  relationship  between 
independent  and  dependent  variables.  Significant  differences  were  observed  between  instructional 
arrangement  with  regard  to  navigation  (number  of  sites  contacted  and  number  of  returns  to  the 
homepage).  An  interaction  between  instructional  arrangement  and  homepage  design  was  observed  with 
regard  to  navigation.  No  significant  differences  with  respect  to  attitude  and  performance  were  observed. 
Implications  for  instruction,  navigation,  and  interface  design  are  detailed.  Areas  for  further  research  are 
presented. 


Theoretical  Framework 

The  theoretical  basis  for  this  comparison  is  grounded  in  the  literature  on  cooperative  learning,  computer- 
mediated  instruction,  and  learner  control  within  hypertext  environments.  From  a social  cognition  perspective,  when 
students  interact  with  their  peers  they  are  faced  with  ideas,  explanations  and  information  that  are  inconsistent  with,  or 
contradict,  their  prior  knowledge  and  beliefs.  Confronting  these  inconsistencies  and  contradictions  challenges 
learners'  current  point  of  view,  and  resolving  such  cognitive  conflicts  results  in  cognitive  restructuring  (Bearison, 
1982).  When  students  receive  new  information  from  peers  during  group  work,  they  presumably  add  detail  to  their 
existing  cognitive  schemas,  explore  and  correct  their  misunderstandings.  Fill  in  gaps  in  their  knowledge,  and/or 
reorganize  their  knowledge  structures  (Bearison,  1982;  Piaget,  1963;  Vygotsky,  1978).  During  peer  interaction, 
students  are  exposed  to  new  strategies,  terminology,  and  ways  of  thinking  about  problems;  these  experiences  can 
restructure  their  own  thinking,  which  may  in  turn  affect  their  problem-solving  behavior.  From  a social  modeling 
perspective,  peer  interaction  provides  opportunities  for  imitation  of  successful  problem-solving  behavior.  Both  of 
these  theoretical  perspectives  can  cause  changes  in  students’  levels  of  competence  as  a result  of  peer  interaction  in 
small  groups  (Vygotsky,  1978). 

Cooperative  learning  (CL)  refers  to  instructional  methods  in  which  students  are  assigned  to  work  in  small, 
intentionally  selected  groups  in  which  they  are  responsible  both  for  their  groupmates’  learning  (interdependent  goal 
structures)  as  well  as  their  own  (individual  accountability)  (Mainzer,  Mainzer,  Slavin,  Lowery,  1993;  Stahl,  1994). 
One  of  the  basic  components  of  cooperative  learning  is  a common  concern  or  goal  shared  by  the  group  of  students, 
with  each  "team"  member  having  a specific  objective  to  be  attained  in  cooperation  with  other  group  members.  In  CL, 
each  member  of  the  team,  as  well  as  the  total  group  effort,  is  assessed  in  terms  of  productivity.  Group  success  is 
typically  assessed  in  terms  of  the  supportive,  contributive,  and  shared  efforts  of  each  team  member  toward  the  group 
attainment  of  a final  outcome.  CL  provides  for  individual  differences  by  enabling  members  to  help  one  another  to 
achieve  the  group  goal  (Slavin,  1982).  Research  indicates  that  students  completing  CL  group  tasks  tend  to  have 
higher  academic  test  scores,  higher  self-esteem,  greater  number  of  positive  social  skills,  and  greater  comprehension 
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of  the  content  and  skills  they  are  studying,  than  those  in  traditional  instructional  situations  ( Johnson,  Johnson,  & 
Holubec,  1993;  Johnson,  Johnson  & Smith,  1991;  Slavin,  1991;  Stahl,  1994;  Stahl  and  VanSickle,  1992).  In  most 
CL  situations,  students  are  working  in  groups  of  two  or  more,  mutually  searching  for  understanding,  solutions,  and 
meaning,  and/or  creating  a product.  This  emphasis  on  academic  success/leaming  for  each  individual  as  well  as  for 
the  group  or  team  as  a unit  is  one  feature  that  distinguishes  cooperative  learning  groups  from  other  group  tasks 
(Slavin,  1991;  Stahl,  1994). 

Little  experimental  research  has  been  conducted  on  the  use  of  cooperative  learning  in  higher  education. 
Dansereau  (1988)  examined  the  effect  of  cooperative  learning  strategies  with  university  student  and  found  that 
students  gained  more  from  learning  in  pairs  than  from  working  alone.  Learning  through  interaction  also  transferred 
to  individual  achievement.  Sherman  (1986)  used  cooperative  learning  methods  with  university  students  enrolled  in 
psychology  courses,  finding  no  significant  differences  on  measures  of  achievement.  Students  in  cooperative  settings, 
however,  rated  their  experiences  more  highly  than  those  who  were  not  in  groups.  Davidson  (1990)  reported  similar 
results  with  college  mathematics  students.  Frierson  (1986)  reported  that  nursing  students  studying  cooperatively 
achieved  higher  scores  on  state  nursing  exams  than  a control  group  studying  independently. 

At  the  University  of  California,  Berkley,  enrichment  workshops  were  formed  that  used  cooperative  learning 
techniques  for  in-coming  African-American  mathematics  and  science  majors  (Treisman,  1985).  Grade  point 
averages,  retention  rate  and  graduation  rates  were  higher  for  students  in  the  cooperative  learning  workshops  than  for 
a control  group  of  students.  In  a meta-analysis  of  approximately  100  studies  conducted  with  subjects  at  the  college 
level,  Bligh  (1979)  found  that  students  who  had  in-class  opportunities  to  actively  interact  with  the  instructor  and  with 
each  other  reported  significantly  higher  levels  of  satisfaction  with  their  learning  experience  than  students  in  classes 
taught  exclusively  by  the  lecture  method.  Kulik  and  Kulik  (1979)  reported  similar  findings  in  their  expansive  review 
of  the  literature,  concluding  that  students  involved  in  classes  which  made  use  of  discussion  groups  were  more  likely 
to  develop  positive  attitudes  toward  the  subject  matter  of  the  course  (Cooper  et  al,  1990). 

Although,  research  addressing  the  use  of  CL  at  the  college  level  is  somewhat  limited,  results  to  date  are  very 
consistent  with  those  reported  in  pre-college  settings  (Cooper,  Prescott,  Cook,  Smith,  Mueck,  & Cuseo,  1990, 
Johnson  & Johnson,  1989;  Johnson,  Johnson  & Smith,  1992).  Procedural  elements,  which  are  critical  to  the 
effectiveness  and  ultimate  success  of  CL,  include: 

• positive  interdependence  of  group  members  (a  feeling  that  the  goal  cannot  be  accomplished  without  everyone's 
involvement), 

• face-to-face  promotive  interaction  (engaging  each  other  in  an  encouraging,  supportive,  and  useful  manner), 

• individual  accountability  and  personal  responsibility  (individual  evaluation  is  contingent  on  individual  performance), 

• interpersonal  and  small  group  skills  (knowledge  and  ability  to  perform  functional  group  roles)  and  group  processing 
(examining  the  workings  of  the  group  relationship)  (Cooper  et  al,  1990;  Johnson  et  al,  1989,  1992;  Johnson  et  al.,  1991; 
Stahl,  1994). 

Because  of  limited  access  to  computers,  assigning  groups  rather  than  individuals  to  work  at  computers  has 
been  common  practice  in  classrooms  of  all  grade  and  academic  levels  for  some  time.  Becker  (1984)  found  in  his 
national  survey  of  k-12  schools  that  teachers  assigned  students  to  groups  to  work  individually  with  computers  46% 
of  the  time.  Several  studies  addressed  group  work  with  computers,  finding  group  activities  to  facilitate  positive  peer 
interactions  (Fisher,  1984;  Mevarech,  Stem,  & Levita;  Webb,  1984)  and  encourage  social  modeling  (Johnson  & 
Johnson,  1985).  Students  completing  computer-meditated  instruction  in  cooperative  groups  perform  as  well  as  and 
often  better  than  students  working  alone  (Carrier  & Sales,  1987;  Dalton,  Hannafin,  & Hooper,  1989;  Hooper,  1992; 
Johnson,  Johnson,  & Stanne,  1985;  Mevarech,  Silber,  & Fine,  1991;  Shlechter,  1990).  Within  the  past  decade, 
research  in  the  area  of  computer-mediated  instruction  (CMI)  has  demonstrated  the  positive  effects  of  this  method  on 
achievement,  attitudes,  and  learning  (Mevarech,  Stem  & Levita,  1987).  In  the  early  1980’s,  Kulik,  Bangert,  and 
Williams  (1983,  1986)  reported  that  39  out  48  studies  demonstrated  the  superiority  of  CMI  programs  over  traditional 
instruction  at  the  college  and  secondary  levels  of  education.  As  indicated  in  their  meta-analysis,  CMI  students 
scored  .32  standard  deviation  higher  on  achievement  tests  than  non-CMI  students. 

Cooperative  CMI  involves  the  instructional  use  of  the  computer  combined  with  cooperative  learning  groups 
(Johnson  & Johnson,  1986).  Research  has  not  extensively  investigated  the  impact  of  cooperative  CMI  in  small 
groups  on  students’  social  development,  especially  with  adult  learners.  Studies  involving  highly  structured 
cooperative  learning  settings  have  found  that  students  using  cooperative  CMI  perform  better  on  achievement  tests 
than  students  using  CMI  individually  (Carrier  & Sales,  1987;  Johnson  & Johnson,  1985).  As  both  computer-assisted 
instruction  and  cooperative  learning  have  demonstrated  positive  effects  on  student  attitude  and  achievement  (Kulik  et 
al,  1983,1986;  Slavin,  1980),  the  two  strategies  when  combined  hold  potential  for  a robust  instructional  strategy 
(Kacer,  Rocklin,  & Weinholtz,  1991). 

The  presentation  of  the  information  in  a computer-mediated  setting  can  greatly  influence  student 
achievement.  Early  promoters  of  hypertext  hailed  it  as  a revolution  in  learning,  allowing  one  to  combine  linear  study 
with  self-exploration  and  chance  discovery.  Some  researchers  believe  that  the  organization  of  human  memory  is  in 
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the  form  of  associations  between  concepts  and  ideas,  much  like  that  of  a hypertext  document  (Beasley  & Lister, 
1992).  The  use  of  hypertext  documents  may  improve  comprehension  and  learning  by  focusing  on  the  relationship 
between  concepts  and  ideas  instead  of  isolated  facts  (Johassen,  1988;  Kearsley,  1988).  Several  studies  lend  support 
to  this  theory  by  suggesting  that  comprehension  can  be  improved  when  the  material  to  be  learned  is  presented  in  such 
a way  that  the  learner  has  some  degree  of  control  over  its  sequence  (Friend  & Cole,  1990).  Due  to  its  flexibility,  and 
the  degree  to  which  learners  are  forced  to  take  an  active  role  in  their  own  learning,  hypertext  is  attractive  to 
educators  who  view  the  ideal  learner  as  self-motivated  and  self-directed  (Shin,  Schallert  & Saveny,  1994).  Learner 
control  is  a commonly  used  in  reference  to  instructional  design  that  allows  a student  to  make  instructional  decisions 
(Park,  1991;  Shin  et  al , 1994).  Reigeluth  and  Stein  (1983)  suggested  that  instructional  effectiveness  and  efficiency 
improve  as  learner  control  increases.  Learner  control  nurtures  system  independence  by  encouraging  students  to 
select  and  manipulate  instruction  according  to  their  needs  and  stimulating  them  to  invest  greater  mental  effort  in  a 
task  (Federico,  1980;  Salomon,  1983,  1985). 

Learner  control  has  not  been  strongly  associated  with  situations  in  which  students  work  alone  (Carrier, 

1984;  Hannafin,  1984;  Milheim&  Martin,  1991;  Steinberg,  1977,  1989).  Learner  control  appears  to  be  most 
effective  when  prior  knowledge  is  high  (Hooper,  Temiyakam,  & Williams,  1994).  However,  as  indicated  by  Carrier 
(1984),  students  given  learner  control  may  not  always  make  "good”  choices.  Learner  control  with  advisement  has 
been  recommended  when  designing  instruction  delivered  by  computers  (Carrier,  1984;  Johansen  & Tennyson,  1983; 
Santiago  & Okey,  1992). 

Several  advisement  strategies  designed  for  improving  students*  choices  under  student  control  have  been 
identified  (Coorough,  1991;  Goetzfield  & Hannafin,  1985;  Holmes,  Robson  & Steward,  1985;  Johansen  & 

Tennyson,  1983;  Laurillard,  1984).  Studies  exploring  the  effects  of  advisement  on  learner  control  have  reported  that 
students  who  received  advisement;  a)  achieved  equal  to  higher  scores  on  achievement  tests  when  compared  to  non- 
advisement control  groups  (Holmes  et  al.,  1985;  Johansen  et  al.,  1983;  Tennyson,  1980),  b)  took  longer  to  complete 
lessons  than  students  in  conventional  learner  control  conditions,  (Goetzfried  et  al,  1985;  Johansen,  1983),  and  c) 
reported  positive  responses  concerning  advisement  in  learner  control  conditions  (Laurillard,  1984). 

The  specific  effects  of  learner  control  in  cooperative  learning  groups  remain  unknown  (Hooper, 

Temiyakam  & Williams,  1994).  Students  less  knowledgeable  of  computers  may  function  effectively  in  learner- 
controlled  environments  when  paired  with  students  who  are  more  knowledgeable.  More  able  partners  will  provide  a 
model  of  effective  learning  that  less  knowledgeable  students  can  emulate.  Coupled  with  advisement  in  the  learner 
control  condition,  these  students  will  show  marked  achievement.  Alternatively,  dominate  partners  may  impose  their 
intentions  on  the  group,  which  may  have  a negative  effect  for  other  members.  In  this  situation,  some  students  may 
benefit  more  from  working  individually  than  in  groups. 

The  purpose  of  the  present  study  is  to  examine  the  effects  of  instructional  grouping  and  internet  homepage 
format  on  student  navigation  in  a network  hypertext  environment,  attitude,  and  performance. 

Methods  and  Techniques 

Fifty  undergraduate  students  (32  female,  18  male)  enrolled  in  an  education  survey  class  at  a large 
southwestern  community  college  served  as  subjects.  Students  ranging  in  age  from  eighteen  to  forty-two  where 
randomly  assigned  to  one  of  four  treatment  conditions;  cooperative  computer-mediated  instructional  groups  with 
navigational  advisement,  cooperative  computer-mediated  instructional  groups  without  navigational  advisement, 
individual  computer- media  ted  instruction  with  navigational  advisement,  and  individual  computer- media  ted 
instruction  without  navigation  advisement.  All  students,  regardless  of  assigned  treatment  received  the  same 
instructional  content  consisting  of  a twenty  minute  introductory  lesson/demonstration  on  the  basic  elements  of 
navigating  through  internet-based  web  sites.  Following  instruction,  each  subject  was  assigned  randomly  to  e a 
treatment  condition  (advisement  or  no  advisement).  The  screen  design  of  the  initial  homepages  differed  in  type  of 
advisement  provided  (advisement  vs.  no  advisement).  In  the  advisement  condition  students  viewed  a homepage  with 
directive  advisement  (Gleason,  1986),  which  included  written  suggestions  for  the  student  on  how  to  navigate  the 
lesson  and  a brief  description  of  the  hypertext  links  contained  in  the  page.  The  no-advisement  condition  consisted  of 
a home-page  containing  a list  of  hypertext  links.  The  specific  links  provided  on  both  pages  were  identical.  Both 
pages  were  designed  using  the  fundamental  design  principals  and  visual  factors  outlined  by  Faiola  & DeBloomis 
(1988)  concerning  fonts,  spacing,  and  basic  graphic  elements.  Students  were  asked  to  write  a brief  essay  detailing 
resources  and  applications  available  for  students  with  special  needs.  They  were  provided  with  access  to  web-based 
search  tools  as  a means  of  gathering  information  in  support  of  their  essay.  Students'  essays  were  evaluated  by 
teachers  enrolled  in  a post  baccalaureate/masters  teacher  training  program.  Writing  samples  were  assessed  via  a 
rubric  adapted  from  the  Arizona  Student  Assessment  Program. 

After  completion  of  the  task  students  were  asked  to  complete  a questionnaire  focusing  on  their  opinions  and 
experiences  related  to  computer- mediated  activities  (Love,  1969) 

Dependent  variables  were  navigation,  attitude,  and  student  performance.  Navigation  was  assessed  by  1)  the 
number  of  links  accessed  by  the  student  or  group  and  2)  the  number  of  times  each  student  returned  to  the  main 
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homepage.  The  network  interface  software  recorded  all  hits  and  returns  to  a data  file.  Attitude  was  assessed  through 
student  responses  to  an  attitude  questionnaire.  Quality  of  the  written  product  was  indicated  by  score  for  quality  of 
writing  as  assigned  by  peer  evaluators.  The  independent  variables  in  this  study  were  1)  instructional  format  and  2) 
advisement.  A two  (instructional  format:  cooperative  groups  vs.  individual  instruction)  x two  (advisement: 
advisement  vs.  no-advisement)  experimental  design  was  constructed  to  assess  impact  of  each  level  of  independent 
variable  on  the  dependent  variables. 

Results  and  Conclusions 
Educational  Importance 

Classical  statistical  procedures  were  employed  to  investigate  the  hypotheses.  Analysis  of  Variance 
(ANOVA)  procedures  were  applied  to  each  dependent  variable  across  levels  of  independent  variable.  All  tests  of 
significance  were  evaluated  at  the  p=.05  level.  The  significant  positive  results  in  attitude  among  subjects  in  the 
group  condition,  and  lack  of  difference  found  in  student  performance  on  the  writing  task  across  treatments,  is 
consistent  with  the  literature  reviewed.  Student  indicate  that  interdependence,  and  the  sharing  of  a common  goal  or 
outcome  was  present  in  most  groups.  Previous  research  indicates  that  pairing  students  based  on  ability,  such  as  low- 
ability  students  paired  with  high-ability  students,  will  show  a greater  effect  on  student  achievement  outcomes  Hooper 
and  Hannafin  (1988).  In  the  present  study,  students  were  randomly  assigned  to  instructional  based  merely  on 
chance,  and  no  other  factors  were  involved  in  the  placement.  Significant  findings  of  the  present  study  involve  student 
navigational  choices  in  a network  hypertext  environment.  The  interaction  between  the  independent  variables  in 
regards  to  the  number  of  unique  sites  visited  indicates  that  students  in  cooperative  learning  groups  across  both  levels 
of  homepage  format  visited  significantly  more  resource  sites  than  students  assigned  to  individual  learning  conditions. 
The  results  of  this  study  indicate  a critical  need  for  research  to  continue  in  this  area.  Curriculum  designers  need  to  be 
aware  of  the  influences  different  settings  have  on  students  working  in  network  hypertext  environments,  this  study 
lays  the  ground  work  for  this  subject  to  be  addressed. 

Discussion 

The  most  intriguing  findings  of  the  present  study  relate  to  the  impact  of  homepage  format  and  instructional 
grouping  on  student  navigation  in  the  World  Wide  Web.  The  interaction  between  the  levels  of  independent  variables 
with  regard  to  the  number  of  unique  URLs  visited  indicates  that  students  in  cooperative  learning  groups  across  both 
levels  of  homepage  format  visited  significantly  more  resource  cites  than  students  assigned  to  individual  learning 
conditions.  Research  geared  towards  the  question  of  whether  these  students  acquired  more  knowledge  by  viewing 
more  homepages  is  yet  to  be  adequately  addressed.  One  explanation  for  this  finding  could  be  that  students  who  work 
individually  lose  interest  in  the  assignment  at  a faster  rate  than  students  working  in  groups.  Conversely,  students 
working  individually  may  be  spending  more  quality  time  on  each  homepage  they  visit  than  subjects  in  the 
cooperative  learning  group. 

The  significant  positive  results  in  attitude  among  subjects  in  the  group  condition  and  lack  of  difference 
found  in  student  performance  on  the  writing  task  across  treatments  is  consistent  with  previous  research.  Although  no 
significant  differences  were  observed  between  groups  in  performance,  cooperative  learning  groups  did  not  appear  to 
have  any  educational  drawbacks.  The  high  attitude  scores  indicated  by  the  student  questionnaires  point  to  a potential 
advantage  of  group  learning  — it  permits  the  effective  instruction  of  at  least  twice  as  many  students  as  does 
instruction  involving  one  person  assigned  to  one  machine.  It  is  difficult  to  determine  why  there  was  not  more  of  a 
difference  between  groups  on  quality  of  work.  Previous  research  indicates  that  pairing  students  based  on  ability, 
such  as  low-ability  students  paired  with  high-ability  students,  will  show  a greater  effect  on  student  achievement 
outcomes  (Hooper  and  Hannafin,  1988).  In  the  present  study,  subjects  were  randomly  assigned  to  instructional 
based  merely  on  chance,  and  no  other  factors  were  involved  in  the  placement.  If  subjects  had  been  assigned  to 
groups  containing  learners  of  varying  levels  of  knowledge,  perhaps  a difference  would  have  been  found.  Also  the 
fact  that  writing  samples  were  not  first  tested  for  sensitivity  towards  the  measures  can  be  another  reason  for  the 
failure  to  find  differences  in  student  performance. 

Diana  and  White  (1994)  suggest  by  combining  three  "high  impact  technologies";  computer  supported 
collaborative  (or  cooperative)  work,  computer-based  learning,  and  the  use  of  (multimedia)  data  bases  for  educational 
purposes,  a kind  of  educational  "Superinterface"  will  evolve  and,  in  turn,  bring  dramatic  changes  in  patterns  of 
knowledge  transfer  in  education.  Curriculum  designers  need  to  be  aware  of  the  influences  educational  and 
instructional  settings  have  on  learners  working  in  network  hypertext  environments  such  as  the  Internet.  Instruction 
need  no  longer  be  restricted  to  the  classroom;  it  may  be  extended  to  the  virtual  classroom  or  a virtual  environment  in 
which  learning  can  take  place.  A class  is  made  up  a virtual  community  of  learners.  Time  becomes  a variable  instead 
of  merely  a fifty-minute  lecture.  In  a distributed  learning  environment,  the  entry  point  may  vary  with  student 
preparation.  The  exit  will  change  depending  on  the  depth  of  mastery  required.  The  length  of  the  learning  activity 
adjusts  to  fit  the  learner’s  needs,  schedule  and  educational  goals.  Successfully  achieving  a distributed  learning 
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environment  requires  the  use  of  information  technology  and  networked  delivery  of  instruction  through  both 
synchronous  and  asynchronous  communication.  To  create  distributed  learning  environments,  institutions  must 
address  both  human  and  technological  issues  through  planning,  support  from  the  community  as  a whole,  and 
innovative  architecture.  The  data  on  student  navigation  through  the  Internet  has  yet  to  be  examined  fully  in  the 
literature.  The  present  study  establishes  ground  work  for  future  research  targeting  specific  questions  which  may 
provide  an  empirical  basis  for  development  effective  learning  tools  in  computer  mediated  network  communication 
environments. 
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Abstract 

The  purpose  of  this  study  was  to  investigate  and  compare  the  level  of  initial  and  long-term  retention  of  frog 
internal  anatomy  among  seventh-grade  students  using  an  interactive  CD  tutorial , a desktop  microworld, 
and  conventional  frog  dissection.  Students'  anxiety  toward  science  was  also  compared  across  the  three 
treatment  groups  and  between  genders.  Additional  data  on  the  students'  preferred  learning  style  were 
used  to  explore  possible  interaction  effects  with  their  respective  instructional  activity. 

Subjects  participating  in  the  study  were  all  seventh-grade  students  in  one  junior-high  school,  numbering 
280  in  total.  Classes  were  randomly  assigned  to  one  of  three  modes  of  instruction.  The  conventional 
treatment  was  traditional  physical  dissection  using  a preserved  frog  specimen  and  lab  dissection  tools. 
The  CD-tutorial  treatment  was  the  interactive  tutorial  Digital  Frog  from  Digital  Frog  International.  The 
microworld  treatment  was  an  environment  composed  of  Operation  Frog  on  CD  supplemented  with  other 
programs  and  resources  to  provide  additional  avenues  for  learning.  Data  collection  and  testing  occurred 
prior  to  treatment,  one  day  after  treatment,  and  three  months  after  treatment. 

Data  analysis  showed  mixed  results  for  all  measures  taken.  Differences  in  achievement  favoring  the 
conventional  treatment  from  pretest  to  both  posttests  appear  to  have  leveled  out  somewhat  over  time. 
Although  anxiety  levels  declined  for  both  genders  after  treatment,  females  reported  significantly  higher 
science  anxiety  than  males  both  before  and  after  treatment.  There  appears  to  be  a relationship  between 
treatment  and  gender  in  terms  of  effect  on  science  anxiety. 

For  all  three  measures  taken  - pretest,  immediate  posttest  and  delayed  posttest  — no  significant  difference 
in  achievement  by  learning  style  was  observed.  Learning  style  alone  does  not  appear  to  be  related  to 
achievement  in  this  study.  However,  the  interaction  between  learning  style  and  treatment  was  significant 
in  some  cases.  In  looking  at  achievement  defined  as  gain  scores  among  the  three  achievement  measures, 
some  cases  within  the  microworld  treatment  proved  to  be  significant. 

Introduction 

This  study  was  bom  from  the  familiar,  frustrating  scene  of  physical  frog  dissection,  which  occurs  annually 
in  school  systems  everywhere,  and  in  this  particular  case,  a suburban  junior  high  school.  The  problem  facing  the 
biology  teachers  was  to  find  an  alternative  means  of  instructional  delivery  that  would  yield  substantially  the  same 
cognitive  knowledge  development  in  the  students,  help  address  the  declining  frog  population,  address  the  issue  of 
science  anxiety  among  students  at  the  junior  high  level,  and  accommodate  the  learning  styles  of  the  students 

Despite  the  long  history  of  dissection  in  biology  coursework,  it  has  become  controversial  (Kinzie,  Foss,  & 
Powers,  1993;  Langley,  1991;  Orlans,  1988;  Strauss  & Kinzie,  1991).  As  one  result,  the  Florida  legislature  passed  a 
bill  in  1988  protecting  the  rights  of  students  who  do  not  wish  to  participate  in  dissection. 

To  explore  potential  new  approaches,  it  was  first  necessary  to  look  at  the  various  alternatives  available  such 
as  anatomical  models,  charts,  laserdiscs,  reference  books,  advanced  coloring  books,  computer  programs,  and 
interactive  Internet  sites.  Secondly,  it  was  necessary  to  select  appropriate  alternative  means  of  instruction  for  this 
grade  level  and  the  depth  of  coverage  desired.  Although  there  are  many  alternatives  to  dissection,  the  extent  to 
which  computer  models,  videotapes  or  other  non-animal  material  can  replace  dissection  entirely  in  schools  remains 
open  to  question  and  appears  to  have  had  little  impact  on  animal  use. 

Many  of  the  alternatives  are  faulted  for  their  low  levels  of  realism  or  for  limited  opportunities  for  active 
student  participation  (Strauss  & Kinzie,  1991).  Whether  or  not  the  alternatives  are  interactive  appears  to  be  an 
important  factor  in  ensuring  success  (Strauss  & Kinzie,  1994).  With  this  in  mind  during  our  search  for  a suitable 
alternative  to  an  actual  frog  dissection,  we  examined  various  forms  of  instructional  material  involving  computers  as 
the  delivery  medium  because  of  their  inherent  constructivist  and  hands-on  elements,  which  should  more  closely 
parallel  the  essence  of  an  actual  dissection  activity.  These  materials  fell  into  one  or  more  of  the  following  general 
categories  (Piskurich,  1993):  hypertext,  hypermedia,  simulation,  multimedia,  virtual  reality,  and  microworld. 

It  soon  became  clear  that  no  one  delivery  system  existed  that  would  cover  the  subject  matter  needed,  at  the 
grade  level  needed,  or  with  a constructivist  hands-on  approach  to  parallel  that  of  an  actual  dissection.  It  also  became 
clear  that  there  had  been  few  efforts  to  evaluate  the  learning  effectiveness  of  alternatives  to  dissection  (e.g.,  Langley, 
1991). 
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The  team  decided  to  use  two  alternative  forms  of  frog  dissection  within  this  study  in  order  to  look  at  the 
supposed  benefit  of  a hands-on  approach  to  learning.  The  selected  alternatives  were  the  CD  tutorial,  Digital  Frog , 
and  a desktop  microworld  with  Operation  Frog  providing  the  hands-on  element  of  the  dissection  process.  It  became 
necessary  to  combine  elements  of  each  delivery  system  category  to  form  a microworld  of  frog  dissection  in  which  to 
immerse  the  students.  This  micro  world  consisted  of  the  multimedia  simulation  Operation  Frog  CD,  digitized 
movies  of  the  various  stages  of  frog  dissection,  3D  movies  of  frog  anatomy  and  human  anatomy,  sounds,  and 
digitized  still  images. 

Problem  of  the  Study 

With  the  richness  of  computer-based  microworlds  come  many  questions  as  to  how  these  environments 
compare  with  actual  reality.  Little  work  has  been  done  in  comparing  the  two  “realities”  with  regard  to 
comprehension  and  recall  of  material;  whether  all  learners  or  only  those  with  certain  learning  styles  can  follow  the 
conceptual  model;  how  the  classroom  teacher  can  harness  the  power  of  these  computer-based  environments;  and  how 
students  and  teachers  interact  with  a microworld.  Even  less  work  has  been  done  specifically  in  the  areas  of 
alternatives  to  physical  dissection  and  their  impact  on  cognitive  development  gains,  long-term  retention,  science 
anxiety,  and  differing  learning  styles. 

Related  Literature 

There  have  been  calls  for  additional  research  to  identify  the  educational  setting,  types  of  students,  and  the 
areas  of  education  that  will  benefit  most  from  the  inclusion  of  hypermedia  (Lamb,  1991;  Jonassen,  1989;  Spiro  & 
Jehng,  1990).  Questions  include  whether  this  type  of  presentation  improves  comprehension  and  recall  of  material, 
whether  all  learners  or  only  those  with  a "hyper-mind"  can  follow  the  conceptual  model  (Maurer,  1993),  and  how  the 
classroom  teacher  can  harness  the  power  of  hypermedia  systems  (Heller,  1990).  In  hypermedia,  elements  of 
constructivist  theory  combine  to  accommodate  the  evolving  nature  of  knowledge  by  encouraging  students  to  engage 
in  a continuing  search  for  improved  understanding  (Resnick,  1989;  Bednar,  et  al.,  1991;  Black  & McClintock,  1995; 
Lebow,  1993;  Wilson,  Teslow,  & Os man-Jo uchoux,  1995;  Duffy  & Jonassen,  1992;  Wilson,  et  al.,  1995; 
Cunningham,  Duffy,  & Knuth,  1993).  Hypermedia  microworlds  designed  in  such  a way  as  to  give  users  exploratory 
experiences  within  a carefully  controlled  range  of  concepts  and  principles  offer  a practical  compromise  between 
instructivism  and  constructivism  (Spiro  & Jehng,  1990;  Papert,  1980;  Rieber,  1996). 

Contradictions  fill  the  literature  concerning  the  use  and  effectiveness  of  computer-based  instructional 
simulations  (Carroll,  1982;  J.  A.  Kulik,  1994;  Mills,  Amend,  & Sebert,  1985).  More  and  better  simulations  might  be 
needed  to  influence  student  examination  performance.  Extensive  research  is  needed  on  simulation  design  and  use 
(Thomas  & Hooper,  1991). 

Students  struggle  when  they  try  to  learn  in  ways  that  aren’t  natural  or  easy  for  them.  Effective  education 
depends  upon  a sound  match  between  characteristics  of  the  student  and  characteristics  of  the  programs  and  persons 
the  student  encounters  (Messick,  1976).  Although  numerous  studies  have  examined  learning  style  (Helm,  1990; 
Ewing  & Yong,1992)  and  numerous  others  have  explored  CAI  (Wang  & Sleeman,  1993;  Wise  & Okey,  1983),  there 
has  been  much  less  research  conducted  pertaining  to  the  relationships  between  CAI  and  learning  preferences  (Ester, 
1994-1995). 

Knowledge  acquisition  has  been  used  to  contrast  the  effectiveness  of  different  instructional  methods 
(Johnstone  & Sleet,  1994,  Moore  & Miller,  1996,  Grieve,  1992).  The  ease  with  which  something  is  retrieved  from 
long-term  memory  is  directly  proportional  to  how  well  it  was  stored  in  the  first  place  (Baddeley,  1976;  1990; 
Ormrod,  1995).  This  simply  indicates  that  when  re-testing  takes  place,  some  material  is  no  longer  accessible.  The 
question  arises  of  whether  hands-on  technology-based  activities  enhance  learning  by  reinforcing  cognitive 
knowledge  and  improving  retention  (Korwin  & Jones,  1990;  Harrison,  1995). 

Researchers  have  defined  student  attitude  toward  science  as  the  student’s  feelings  of  like  or  dislike  toward 
science  (LaForgia,  1988;  Koballa,  Crawley  & Shrigley,  1990;  Atwater,  Wiggins  & Gardner,  1995).  The  question  is 
to  whether  activity-oriented  science  instruction  can  help  develop  favorable  attitudes  toward  science.  Teaching 
strategy  is  related  to  student  anxiety  and  performance  levels  (Westerback,  1982).  Highly  anxious  elementary  and 
junior-high  students  favored  a less  directive  environment,  while  their  less  anxious  counterparts  favored  stronger 
direction.  Learning  environments  need  to  be  created  where  students  of  different  personalities  and  learning  aptitudes 
can  be  successful,  regardless  of  instructional  mode,  because  students  do  not  learn  equally  well  from  all  modes  of 
instruction. 

Purpose  of  the  Study 

The  purpose  of  this  study  was  to  investigate  and  compare  the  level  of  initial  learning  and  also  long-term 
retention  of  the  frog’s  internal  anatomy  between  seventh-grade  students  using  an  interactive  CD  tutorial,  a desktop 
microworld,  and  conventional  frog  dissection.  Students’  anxiety  toward  science  was  also  compared  across  the  three 
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treatment  groups.  Additional  data  were  collected  on  the  students*  preferred  learning  modality  to  explore  possible 
relationships  with  the  level  of  learning  or  long-term  retention  and  their  respective  instructional  treatments. 

Subjects 

The  population  used  within  this  study  was  comprised  of  354  seventh-grade  students  in  fourteen  classes  in  a 
suburban  school  district.  Of  the  mostly  Caucasian  students,  181  were  boys,  and  173  were  girls.  Due  to  attrition 
and/or  absence  during  one  or  more  phases  of  the  data  collection,  74  students  whose  data  were  incomplete  were 
excluded  from  the  analysis,  reducing  the  subject  pool  to  280  subjects,  142  males  and  138  females.  None  of  the 
students  had  had  any  instruction  in  frog  dissection  prior  to  the  study. 

Research  Design 

A quasi-ex perimental  research  design  was  used,  specifically  the  nonequivalent  control-group  research 
design  (Gall,  Borg,  & Gall,  1996),  because  the  research  participants  could  not  be  randomly  assigned  to  the 
experimental  and  control  groups  due  to  the  naturally  occurring  environment.  Therefore,  intact  classes  were  randomly 
assigned  to  treatments.  All  subjects  took  a pretest,  a posttest,  and  a delayed  posttest.  Except  for  random  assignment, 
the  steps  involved  in  this  design  are  the  same  as  for  the  classic  pretest-posttest  control-group  experimental  design. 

Variables 

The  independent  variable  or  treatment  was  the  method  used  to  deliver  instruction  on  frog  dissection.  The 
dependent  variables  were  the  achievement  scores  on  the  pretest,  posttest,  and  delayed  posttest;  scores  on  a science 
anxiety  instrument  administered  at  the  start  and  end  of  the  experiment;  and  the  preferred  learning  style  of  the 
students. 

Treatments 

The  three  treatments  used  within  this  study  consisted  of  two  experimental  treatments,  a desktop  microworld 
and  an  interactive  tutorial,  and  conventional  physical  frog  dissection.  The  media  for  creating  a desktop  microworld 
and  an  interactive  tutorial  of  a frog  dissection  had  to  be  readily  available,  relatively  easy  to  use  and  able  to  run  on 
existing  platforms  within  an  average  educational  setting.  After  careful  consideration  of  existing  media  for  frog 
dissection,  Digital  Frog  from  Digital  Frog  International  was  chosen  for  the  interactive  tutorial.  However,  a 
combination  of  programs  had  to  be  assembled  to  create  a desktop  microworld  environment. 

Digital  Frog  Interactive  CD-ROM 

Digital  Frog  is  a commercial  interactive  CD-ROM  that  incorporates  full-motion  video,  animations,  sounds, 
narration,  in-depth  text,  full  color  photographs  and  a comprehensive  workbook  in  three  modules  --  Dissection; 
Anatomy;  and  Ecology.  The  dissection  module  uses  a tutorial  approach  with  the  dissection  proceeding  in  a step  by 
step  presentation  manner.  Users  can  access  multimedia  files  to  view  a frog  dissection  being  performed,  but  do  not 
actually  “perform”  a dissection  themselves. 

Desktop  Micro  world 

The  desktop  microworld  environment  was  built  around  the  commercial  product  Operation  Frog  on  CD  with 
other  programs  serving  as  auxiliary  forms  of  instructional  delivery.  Operation  Frog  contains  a tutorial  simulation  of 
the  dissection  of  a frog,  male  or  female.  Unlike  Digital  Frogy  Operation  Frog  allows  students  to  actually  perform 
the  dissection,  not  just  view  it.  However,  unlike  a real  dissection,  students  also  can  reconstruct  the  frog.  Using 
graphical  surgical  scissors,  a probe,  forceps,  and  a magnifying  lens,  students  probe  and  snip  body  organs,  remove 
organs  to  the  examination  tray,  and  investigate  frog  body  systems  close-up. 

To  help  compensate  for  some  of  the  limitations  of  Operation  Frog  and  to  build  a stronger  microworld 
environment,  the  students  were  given  access  to  three  additional  modes  of  instruction: 

• 3D  QuickTime  movies  of  the  frog  at  various  stages  of  dissection,  created  locally  and/or  downloaded  to  each  computer’s  hard 
disk  prior  to  the  treatment  from  the  Internet  site  Virtual  Frog,  [http://george.lbl. gov/ITG.hm.pg.docs/dissect/info.html] 

• QuickTime  movies  of  the  stages  of  dissection  being  performed,  downloaded  to  each  computer’s  hard  disk  prior  to  the  dissection 
activity  from  the  Internet  site  NetFrog.  [http://curry.edschool.Virginia.EDU/  go/frog/] 

• Digitized  pictures  and  sounds  downloaded  from  various  other  sites  on  the  Internet. 


These  components  were  accessible  to  students  at  appropriate  times  during  the  dissection  process  as 
indicated  on  their  worksheets  by  icons  matched  with  file  names,  which  had  been  arranged  at  the  top  of  the  Apple 
menu  for  ease  of  access. 
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Conventional  Frog  Dissection 

The  conventional  physical  dissection  method  was  used  as  the  control  treatment  within  this  study.  It  was  the 
traditional  method  of  providing  a preserved  specimen  of  a frog  for  every  two  students  within  the  class.  Students 
used  traditional  dissection  trays  and  related  tools.  The  dissection  activity  was  conducted  in  the  science  classrooms, 
which  contain  lab  tables  for  science  activities.  Teachers  guided  students  through  the  dissection  process  aided  by 
worksheets  given  to  the  students. 

Lab  Worksheets 

Lab  worksheets  were  composed  of  the  “Frog  Dissection  Laboratory  Investigation”  from  the  Prentice  Hall 
Life  Science  Laboratory  Manual  as  well  as  material  used  by  the  science  team  over  the  course  of  several  years.  The 
content  and  questions  asked  remained  the  same  for  all  subject  groups.  However,  the  directions  given  within  the 
worksheets  reflected  the  individual  treatments  within  the  study. 

The  lab  worksheets  given  to  the  control  group  performing  the  conventional  dissection  included  directions 
for  performing  the  physical  dissection.  The  lab  worksheets  given  to  subjects  in  the  experimental  desktop  microworld 
contained  additional  directions  on  where  to  go  on  the  computer  for  certain  activities.  These  directions  were  in  the 
form  of  symbols,  e.g.,  a movie  camera  symbol  followed  by  a movie  title  for  accessing  the  QuickTime  movies,  a 
camera  followed  by  an  image  file  name  for  accessing  digitized  images,  etc.  The  lab  worksheets  for  the  subjects 
using  the  interactive  tutorial,  Digital  Frog , contained  directions  for  getting  started  in  the  program. 

The  format  of  the  lab  worksheets  was  designed  to  be  easy-to-follow  and  to  allow  the  students  to  complete 
the  investigation  largely  on  their  own,  leaving  the  instructor  free  to  provide  necessary  help  to  individuals  or  groups 
of  students.  The  lab  worksheets  also  asked  for  observations  after  the  procedure  at  each  level  of  dissection  and  asked 
students  to  draw  conclusions  that  encouraged  them  to  use  critical-thinking  skills. 

Procedures 

The  study  was  conducted  within  the  prescribed  seventh  grade  biology  curriculum  following  the  completion 
of  the  unit  on  cold-blooded  vertebrates,  which  is  the  last  unit  curriculum  conducted  each  school  year.  Students  had 
performed  dissections  on  earthworms  earlier  in  the  year  and  were  familiar  with  dissection  instruments  and  general 
lab  procedures  and  safety  precautions. 

Each  of  the  fourteen  intact  science  classes  involved  in  the  study  was  randomly  assigned  an  instructional 
delivery  system  subject  to  scheduling  constraints  (each  teacher  was  to  have  at  least  one  class  of  each  treatment  to 
help  control  for  bias)  and  availability  of  conventional  or  computer  lab  space.  This  process  resulted  in  four  classes 
using  the  interactive  tutorial,  six  classes  using  the  desktop  microworld,  and  four  classes  using  the  conventional 
dissection  method.  No  class  exceeded  30  students.  Prior  to  the  actual  dissection  activity,  teachers  received  training 
in  manipulating  the  computer  treatments,  using  the  same  lab  worksheets  as  the  students  would  use. 

The  students  in  the  experimental  sections  conducted  their  alternative  dissection  activity  using  the  interactive 
tutorial  or  desktop  microworld  in  the  auxiliary  computer  lab  at  the  school.  This  lab  contains  15  Macintosh 
computers,  which  necessitated  the  partnering  of  subjects,  two  to  a computer.  This  was  appropriate,  as  the  control 
group  was  also  partnered,  two  to  a dissection  pan,  as  they  historically  had  been  grouped  within  the  school. 

For  all  groups  the  dissection  took  place  over  a span  of  four  consecutive  days  with  each  class  having  a 43- 
minute  instructional  time  period  each  day.  The  testing,  preparation  and  closure  took  place  during  three  class  periods. 
All  achievement  tests  used  the  same  instrument.  The  pretest  and  preparation  occurred  on  the  day  before  the 
dissection  activity  began.  The  immediate  posttest  and  closure  occurred  on  the  day  following  the  dissection  activity. 
The  class  period  for  administration  of  the  delayed  posttest  was  three  months  following  the  dissection  activity,  during 
the  opening  days  of  the  following  school  year.  The  science  anxiety  survey  was  administered  in  each  class  two  days 
prior  to  the  start  of  the  dissection  activity  to  establish  a baseline  for  comparison  and  then  again  on  the  second  day 
following  the  conclusion  of  the  dissection  activity. 

Data  Collection  / Instruments 

The  three  regular  science  teachers  conducted  the  classes  and  graded  the  pretest,  posttests,  and  lab 
worksheets.  They  also  administered  the  science  anxiety  survey.  The  study  skills  teacher  at  the  school  had  previously 
conducted  the  learning  style  self-assessment. 

The  43-item  science  anxiety  instrument  was  adapted  from  one  developed  for  high  school  age  students 
(Wynstra  & Cummings,  1993).  The  first  35  items  asked  students  to  indicate  how  nervous  they  would  feel  if  they  had 
to  do  each  of  the  listed  science-related  activities.  They  responded  using  an  ordinal  scale  from  “a”  (indicating  “Not  at 
all  Nervous”)  through  “e”  (indicating  “Very  Nervous”.)  The  remaining  eight  questions  were  general  questions  such 
as  their  gender  and  their  interest  in  science  related  topics.  The  interest  questions  were  not  used  within  this  study,  as 
the  science  teachers  deemed  them  geared  toward  the  high-school  aged  student. 

Since  the  science  anxiety  instrument  was  originally  targeted  at  high-school  age  students,  the  reliability  of 
the  instrument  for  this  age  group  was  tested  by  administering  the  instrument  prior  to  the  study  to  one  class  chosen  at 
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random  from  the  fourteen  classes  of  subjects.  With  a maximum  possible  value  of  1,  Cronbach's  Alpha  yielded  an 
alpha  level  of  .9005  as  the  reliability  coefficient.  Thus  the  instrument  was  deemed  reliable  for  this  study. 

The  pretest  and  posttest  were  identical  and  were  administered  to  both  the  control  group  and  the 
experimental  groups.  The  test  was  comprised  of  items  from  the  Prentice  Hall  Life  Science  text  for  the  junior-high 
school  student  and  the  lab  worksheets  used  within  the  science  curriculum  for  conventional  dissection.  Items 
consisted  of  multiple-choice,  true-false,  and  matching  questions  that  either  had  already  been  tested  for  this  subject 
area  and  level  using  item-analysis  techniques  and  been  found  to  be  reliable  or  had  been  judged  to  be  reliable  by  the 
science  team  at  the  school  over  the  years. 

Data  Analysis 

Statistical  analysis  of  data  was  completed  using  SPSS  7.5  software.  The  level  of  significance  for  all  tests 
was  set  at  0.05. 

Demographic  data  concerning  the  subjects  were  tabulated  using  frequency  counts.  Because  the  subjects 
could  not  be  randomly  assigned  to  treatments,  the  issue  of  initial  equivalency  among  the  subjects  was  addressed.  A 
pre-test  was  administered  and  when  the  scores  were  analyzed  by  gender  using  a t test  and  by  treatment  group  and 
specific  class  using  analysis  of  variance  (ANOVA),  no  significant  differences  were  found  (see  Table  1 for  pretest 
statistics).  To  address  the  research  hypotheses,  which  focused  on  differences  across  the  three  treatments,  several 
techniques  were  employed.  Achievement  and  retention  were  defined  as  the  change  in  individuals*  scores  on  the 
achievement  test  from  pre-test  to  post-test,  from  post-test  to  delayed  post-test,  and  from  pre-test  to  delayed  post-test. 
Means  and  standard  deviations  were  calculated  for  each  test  by  treatment  to  identify  patterns.  The  “gain”  scores  for 
test  pairs  were  calculated  and  submitted  to  one-way  ANOVAs  to  identify  any  statistically  significant  findings  relative 
to  each  hypothesis.  With  three  treatment  conditions,  further  analysis  of  significant  ANOVAs  was  required  to 
identify  the  individual  pair(s)  that  differed  significantly.  Contrast  tests  were  selected  for  this  aspect  of  the  analysis. 
The  possible  interaction  effect  on  achievement  of  treatment  and  learning  style  of  subjects  was  also  explored. 

Analysis  of  these  data  utilized  descriptive  statistics,  analysis  of  variance  across  learning  styles  within  treatment,  as 
well  as  a factorial  ANOVA  for  the  three  treatments  by  the  seven  identified  learning  styles. 
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Table  1.  Pretest  Scores  by  Gender,  Treatment , and  Class  Section 


N 

Mean 

SD 

Gender 

Male 

142 

21.38 

5.31 

Female 

138 

22.12 

4.36 

Treatment 

CD  Tutorial 

86 

21.97 

4.62 

Microworld 

125 

22.13 

4.66 

Conventional 

69 

20.78 

5.44 

Class 

Ml 

25 

22.72 

4.84 

M2 

26 

22.27 

3.87 

M3 

22 

22.05 

4.61 

M4 

18 

21.78 

5.17 

M5 

23 

23.13 

4.43 

M7 

24 

21.96 

6.00 

SI 

18 

23.11 

3.85 

S2 

22 

20.95 

3.40 

S3 

18 

21.61 

5.13 

S4 

15 

20.20 

7.34 

S5 

17 

23.29 

3.90 

S6 

15 

20.93 

4.45 

W4 

18 

19.33 

5.92 

W5 

19 

19.95 

4.22 

Findings  Pertaining  to  Research  Question  One 

Research  question  one  asked,  What  are  the  relative  effects  on  student  achievement  of  instruction  using  an 
interactive  tutorial,  a desktop  microworld,  and  a conventional  dissection  of  a frog?  Three  hypotheses  proposed  no 
significant  difference  among  the  three  treatment  groups  in  gain  from  pretest  to  posttest,  pretest  to  delayed  posttest  or 
posttest  to  delayed  posttest. 

In  looking  at  the  relationship  between  treatment  and  gain  scores,  significant  differences  were  observed.  The 
mean  gain  from  pretest  to  immediate  posttest  for  students  participating  in  the  conventional  dissection  (Mean  = 1 1.99) 
was  significantly  higher  [t  = 2.97,  p = .007]  than  for  students  in  the  microworld  treatment  (Mean  = 9.78).  The 
conventional  dissection  group  (Mean  = 7.36)  also  showed  a significantly  higher  mean  gain  score  from  pretest  to 
delayed  posttest  than  either  the  microworld  group  (Mean  = 4.24.J  = 4.03,  p = .000)  or  the  CD-tutorial  group  (Mean 
= 4.14,  t = -3.862,  p = .000).  There  was  no  significant  difference  in  long  term  retention  among  treatments  as 
evidenced  by  gain  scores  from  posttest  to  delayed  posttest. 

In  summary  for  the  first  research  question  and  its  three  hypotheses,  the  data  supported  rejecting  the 
hypothesis  of  no  significant  difference  in  achievement  by  treatment  in  the  case  of  gain  from  pretest  to  immediate 
posttest  as  well  as  in  the  case  of  pretest  to  delayed  posttest  scores.  Results  favored  the  conventional  dissection 
treatment.  However,  the  data  did  not  support  rejecting  the  hypothesis  in  the  case  of  gain  from  immediate  posttest  to 
delayed  posttest,  suggesting  a leveling  of  results  over  time.  Regardless  of  the  measure,  students  in  the  conventional 
dissection  treatment  did  outperform  those  in  the  two  experimental  treatments,  which  did  not  differ  significantly  from 
one  another. 

Findings  Pertaining  to  Research  Question  Two 

The  second  research  questions  was  What  are  the  relative  effects  and  the  effects  of  gender  on  students'  level 
of  science  anxiety  of  instruction  using  an  interactive  tutorial,  a desktop  microworld,  and  a conventional  dissection  of 
a frog?  Two  hypotheses  proposed  no  significant  difference  in  science  anxiety  by  treatment  group  or  gender.  The 
science  anxiety  survey  was  given  before  the  start  and  after  the  conclusion  of  the  frog  dissection  unit.  Results  showed 
the  three  treatment  groups  were  not  equivalent  in  anxiety  at  the  beginning.  Contrast  tests  revealed  that  students 
assigned  to  the  conventional  treatment  reported  significantly  lower  science  anxiety  than  students  in  either  the  CD 
tutorial  (t  = 3.1 1,  p = 0.002)  or  the  microworld  treatment  (t  = 2.84,  p = 0.005.)  The  mean  science  anxiety  score  for 
students  in  the  CD  tutorial  group  did  not  differ  from  that  of  the  micro  world  group. 
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These  initial  findings  did  not  change  after  treatment,  and  no  single  treatment  group  showed  a significantly 
different  change  in  anxiety  compared  to  the  other  treatment  groups.  Contrast  tests  showed  that  students  experiencing 
the  conventional  treatment  reported  significantly  lower  science  anxiety  after  treatment  than  students  in  either  the  CD 
tutorial  group  (t  = 2.71,  p = 0.007)  or  the  microworld  treatment  (t  = -3.56,  p < 0.001.)  The  mean  post- treatment  score 
for  students  in  the  CD  tutorial  group  did  not  differ  from  the  microworld  group. 

In  looking  at  the  results  of  the  science  anxiety  measures  by  gender,  some  interesting  patterns  emerged.  A t 
test  showed  that  females  reported  significantly  higher  initial  science  anxiety  than  males  (t  = -3.46,  df  = 278,  p = 
0.001).  Although  anxiety  levels  declined  for  both  genders  after  treatment,  females  continued  to  report  significantly 
higher  science  anxiety  than  males  (t  = -3.36,  df  = 278,  p = 0.001).  Therefore,  the  hypothesis  of  no  significant 
difference  in  anxiety  by  gender  was  rejected  for  both  the  pre-treatment  and  post-treatment  survey. 

Looking  at  differences  in  anxiety  levels  by  treatment  received  and  by  gender,  the  three  treatment  groups 
displayed  varying  results.  Males  and  females  in  the  CD  tutorial  treatment  showed  a significant  difference  before 
treatment  (t  = -2.13,  df  = 84,  p = 0.036)  but  not  after  treatment  (t  = -1.195,  df  = 84,  p = 0.236).  Participation  in  the 
CD  tutorial  treatment  appears  to  have  closed  the  gap  between  the  genders.  This  was  in  contrast  to  the  conventional 
treatment  subjects  who,  although  showing  a general  decline  overall  in  anxiety  levels  after  treatment,  showed  a 
significant  difference  between  genders  both  before  (t  = -2.92,  df  = 67,  p = 0.005)  and  after  treatment  (t  = -2.35,  df  = 
67,  p = 0.022).  Apparently  the  conventional  dissection  was  not  effective  in  reducing  the  disparity  in  anxiety  between 
male  and  female  subjects.  For  the  microworld  group,  anxiety  levels  across  genders  did  not  differ  significantly  before 
treatment.  However,  after  treatment,  the  difference  did  approach  significance  (t  = -1.94,  df  = 123,  p = 0.55),  with 
anxiety  declining  more  amongst  males  than  females. 

In  summary  for  the  second  research  question  and  its  two  hypotheses,  the  data  supported  rejecting  the 
hypothesis  of  no  significant  difference  in  students’  level  of  science  anxiety.  The  three  treatment  groups  were  not 
equivalent  in  anxiety  at  the  beginning  and  this  pattern  held  after  treatment  as  well.  The  data  also  supported  rejecting 
the  hypothesis  in  the  case  of  gender.  Females  consistently  reported  a significantly  higher  science  anxiety  level  than 
their  male  counterparts,  both  before  and  after  treatment.  When  looking  at  differences  in  anxiety  levels  by  treatment 
received  and  by  gender,  the  data  again  supported  rejection  of  the  null  hypothesis.  While  the  means  by  gender  for  the 
CD  tutorial  treatment  and  the  conventional  treatment  were  significantly  different  before  treatment  began,  males  and 
females  in  the  microworld  treatment  did  not  differ  significantly.  After  treatment  males  and  females  in  the  CD  tutorial 
treatment  no  longer  showed  a significant  difference.  For  the  microworld  group,  the  post-treatment  mean  difference 
approached  but  did  not  reach  a level  of  significance,  with  anxiety  declining  more  among  male  students  than  female 
students.  Within  the  conventional  treatment  group,  although  the  level  of  anxiety  lowered  across  genders,  there  was 
still  a significant  difference  between  males  and  females  after  treatment. 

Findings  Pertaining  to  Research  Question  Three 

The  third  research  question  asked  Is  there  a relationship  between  students’  preferred  learning  style  and 
comprehension  gain  scores  as  they  relate  to  the  type  of  instructional  delivery?  Hypotheses  were  that  there  was  no 
significant  difference  in  achievement  by  learning  style  and  no  interaction  between  treatment  and  learning  style. 

When  looking  at  the  relationship  between  students’  preferred  learning  style  and  achievement,  no  significant 
difference  in  achievement  by  learning  style  was  found  for  any  of  the  three  measures  taken  — pretest  [F(5,274)  = 

1.125,  p = 0.347],  immediate  posttest  [F(5,274)  = 0.707,  p = 0.619],  or  delayed  posttest  [F(5,274)  = 0.664,  p = 
0.651].  Learning  style  alone  does  not  appear  to  be  related  to  achievement  in  this  study. 

However,  significant  interactions  were  observed  between  treatment  and  preferred  learning  style  in  terms  of 
achievement.  An  analysis  of  variance  found  the  interaction  between  learning  style  and  treatment  on  the  immediate 
posttest  was  significant  [F(  10,269)  = 2.11,  p = 0.024].  To  identify  the  source  of  this  interaction,  separate  analyses  of 
variance  were  performed  on  the  immediate  posttest  data  by  learning  style  for  each  of  the  three  treatments.  The  results 
for  the  CD  tutorial  treatment  were  not  significant  [F(5,80)  = 0.589,  p = 0.709].  Likewise,  the  results  for  the 
microworld  treatment  were  not  significant  [F(5, 1 19)  = 1.703,  p = 0.139].  However,  the  results  for  the  conventional 
treatment  were  significant  [F(5,63)  = 2.396,  p = 0.047].  To  determine  which  pairs  differed  significantly,  contrast 
tests  were  used.  These  tests  showed  that  the  immediate  posttest  mean  score  for  visual-kinesthetic  learners  in  the 
conventional  treatment  (M  = 26.00)  was  significantly  lower  than  the  mean  score  for  auditory  learners  (M  = 32.63), 
auditory-kinesthetic  learners  (M  = 35.50),  kinesthetic  learners  (M  = 34.18),  and  visual  learners  (M  = 34.73).  The 
computed  t-test  values  are  shown  in  Table  2. 
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Table  2.  T-test  Values  for  Significantly  Different  Means , Immediate  Posttest  by  Learning  Style ; Conventional 
Treatment 


Mean  Pair 

Difference 

T 

DF 

Sie 

VK-A 

-6.63 

2.411 

63 

0.019 

VK-AK 

-9.50 

2.534 

63 

0.014 

VK-K 

-8.18 

2.714 

63 

0.009 

VK-V 

-8.73 

3.026 

63 

0.004 

VK  = visual-kinesthetic 
A = auditory 

AK  = auditory-kinesthetic 
K = kinesthetic 
V = visual 


An  analysis  of  variance  for  the  delayed  posttest  scores  by  treatment  within  learning  style  showed  no 
significant  interaction  [F(  10,269)  = 1.047,  p = 0.404]. 

To  summarize  briefly,  the  hypothesis  of  no  significant  interaction  between  treatment  and  preferred  learning 
style  was  rejected  for  the  immediate  posttest  scores,  but  only  for  the  conventional  dissection  treatment.  Visual- 
kinesthetic  learners  scored  significantly  lower  on  average  on  the  posttest  than  their  peers  who  preferred  either  the 
auditory,  auditory- kinesthetic,  kinesthetic,  or  visual  learning  styles.  No  such  difference  existed  initially,  and  it 
disappeared  again  at  the  delayed  posttest. 

Turning  to  achievement  defined  as  gain  scores,  an  analysis  of  variance  of  the  gains  from  pretest  to 
immediate  posttest  by  learning  style  and  treatment  yielded  a significant  interaction  (Table  3).  Comparable  analyses 
of  gains  from  pretest  to  delayed  posttest  and  from  immediate  to  delayed  posttest  showed  no  significant  interaction. 


Table  3.  Analysis  of  Variance,  Gain  from  Pretest  to  Immediate  Posttest  by  Learning  Style  and  Treatment 


Source 

SS 

DF 

MS 

F 

SiE 

Main  Effects 

141.54 

7 

20.22 

0.850 

0.547 

LS 

101.15 

5 

20.23 

0.850 

0.515 

Treatment 

28.19 

2 

14.095 

0.592 

0.554 

Interaction 

117.64 

10 

44.77 

1.912* 

0.017 

Residual 

6232.86 

262 

23.79 

Total 

8318.84 

279 

29.82 

*p  < .05 


Contrast  tests  revealed  the  significant  interaction  of  learning  style  and  treatment  was  attributable  to  four 
pairings  of  scores  by  learning  style  preferences  within  the  microworld  treatment.  Students  showing  a preference  for 
the  kinesthetic  learning  style  achieved  significantly  lower  gain  scores  (Mean  = 8.24)  than  students  with  a preference 
for  the  auditory  learning  style  (Mean  = 10.84;  t = 2.29,  p = .030).  Gain  scores  for  kinesthetic  learners  were  also 
significantly  lower  than  for  auditory-kinesthetic  learners  (Mean  = 10.67 ,_t  = 2.10,  p = .045)  and  for  auditory-visual 
learners  (Mean  = 14.20,  t = 3.29,  p = .005). 

In  addition,  the  mean  gain  from  pretest  to  immediate  posttest  among  students  exhibiting  a preference  for  the 
auditory-visual  learning  style  (Mean  = 14.20)  was  significantly  higher  [t  = 3.00,  p = .010]  than  the  gain  for  those 
students  preferring  the  visual  learning  style  (Mean  = 9.08).  It  should  also  be  noted  that  the  gain  scores  for  students 
with  a preference  for  the  auditory- visual  learning  style  approached,  but  did  not  quite  reach,  significance  [t  = -2.10,  p 
= .058]  compared  to  students  demonstrating  a preference  for  the  Visual-Kinesthetic  learning  style  (Mean  = 10.30). 

Based  on  the  preceding  analyses,  the  hypothesis  of  no  significant  interaction  between  learning  style  and 
treatment  relative  to  achievement  (defined  as  gain  scores)  was  rejected  for  some  cases  within  the  microworld 
treatment.  Students  who  preferred  the  three  variations  on  auditory  learning  attained  the  greatest  gains,  and  each  of 
the  three  was  significantly  higher  than  the  gain  for  students  with  a kinesthetic  preference.  Students  in  the  auditory- 
visual  group  also  significantly  outperformed  those  who  preferred  the  visual  learning  style. 

In  summary  for  the  third  research  question  and  its  two  hypotheses,  the  hypothesis  of  no  significant 
difference  in  achievement  by  learning  style  preference  could  not  be  rejected  for  any  of  the  three  measures  taken  - 
pretest,  immediate  posttest,  or  delayed  posttest.  Learning  style  alone  does  not  appear  to  be  related  to  achievement  in 
this  study.  However,  the  hypothesis  of  no  significant  interaction  between  treatment  and  preferred  learning  style  was 
rejected  for  the  immediate  posttest  scores  from  the  conventional  dissection  treatment.  Here,  visual-kinesthetic 
learners  scored  significantly  lower  on  average  than  their  peers  who  showed  a preference  for  any  of  the  other  four 
learning  styles.  This  difference  was  not  found  on  the  pretest  and  disappeared  at  the  delayed  posttest. 

When  looking  at  the  interaction  between  learning  style  and  treatment  relative  to  achievement  defined  as 
gain  scores,  the  hypothesis  was  rejected  for  some  cases  within  the  microworld  treatment.  The  greatest  gains  were 
reported  by  students  preferring  the  three  auditory  learning  variations,  with  each  showing  a significantly  higher  gain 


than  students  with  a kinesthetic  preference.  In  addition,  students  in  the  auditory-visual  group  significantly 
outperformed  those  who  preferred  the  visual  learning  style. 

Discussion  and  Conclusions 

A discussion  of  the  results  and  conclusions  based  on  these  results  are  presented  next,  based  on  the  broad 
areas  of  inquiry  defined  by  the  research  questions. 

Relative  Effects  on  Student  Achievement 

In  looking  at  the  relationship  between  instructional  treatment  and  gain  scores  achieved,  significant 
differences  were  observed.  Conventional  dissection  appears  to  generate  greater  learning  than  the  microworld 
treatment  when  looking  at  gains  from  pretest  to  immediate  posttest.  However,  no  difference  was  found  between  the 
conventional  dissection  and  the  CD-tutorial  treatment.  Therefore,  if  the  desired  outcome  is  immediate  gain  of 
knowledge,  the  preferred  delivery  system  would  appear  to  be  either  the  conventional  dissection  or  the  CD-tutorial. 

The  question  is  why  students  within  the  microworld  treatment  scored  significantly  lower  than  both  other 
treatments.  If  the  hands-on  component  is  thought  to  provide  better  encoding  of  information,  then  the  logical 
conclusion  would  be  that  both  the  conventional  and  the  microworld  treatment  groups  would  outperform  the  CD 
tutorial  group  on  the  immediate  post- test.  Perhaps  the  explanation  lies  in  the  findings  of  Thomas  and  Hooper  (1991). 
They  suggested  that  the  effects  of  simulations  are  revealed  not  by  tests  of  knowledge  but  by  tests  of  transfer  and 
application.  Another  explanation  can  also  be  offered  by  looking  at  the  students'  experience  with  the  various  forms  of 
instructional  treatment.  All  students  had  previous  dissection  experience  and  all  students  had  experience  with  CD 
tutorials,  albeit  not  one  in  which  an  entire  science  unit  was  delivered  via  this  method.  However,  the  microworld 
environment  was  completely  new  to  their  range  of  experiences  and  may  have  resulted  in  a lowering  of  their 
immediate  achievement  gain  due  to  unfamiliarity  with  the  format. 

When  looking  at  the  mean  gain  score  over  time,  from  pretest  to  delayed  posttest,  students  in  the 
conventional  dissection  method  showed  significantly  better  retention  than  those  in  either  the  microworld  or  the  CD- 
tutorial  treatment.  This  seems  to  give  the  clear  advantage  to  the  conventional  method  of  dissection,  but  this 
advantage  disappears  when  looking  at  the  gain  scores  from  posttest  to  delayed  posttest.  Here  the  differences 
favoring  the  conventional  treatment  have  disappeared,  as  no  one  treatment  proved  to  be  significantly  better  for 
retaining  knowledge  gained  over  the  long  term  than  either  of  its  counterparts.  In  other  words,  the  knowledge 
decrement  (Grieve,  1992)  was  not  significantly  different  across  treatments. 

When  looking  at  knowledge  retained  over  the  long  term,  this  study's  results  support  the  view  that 
alternatives  to  dissection  are  as  effective  as  the  actual  practice  of  physical  dissection.  This  lends  support  to  the 
National  Association  of  Biology  Teachers  (NABT)  official  policy  supporting  alternatives  to  dissection  and 
vivisection  in  the  biology  curricula  wherever  possible,  as  long  as  those  alternatives  satisfy  the  objectives  of  teaching 
scientific  methodology  and  fundamental  biological  concepts.  In  the  debate  over  animal  dissection  in  the  classroom, 
a number  of  studies  (Downie  & Meadows,  1995;  Leonard,  1989;  Quentin-Baxter  & Dewhurst,  1992;  Strauss  & 
Kinzie,  1991;  Strauss  & Kinzie,  1994;  Kinzie,  Larsen,  et  al.,  1996)  have  attempted  to  find  suitable  alternatives  to 
animal  dissection.  This  study  has  shown  two  forms,  a CD  tutorial  and  a desktop  microworld,  to  be  viable 
alternatives  to  physical  frog  dissection  in  retaining  knowledge  gained  over  the  long  term. 

Relative  Effects  on  the  Level  of  Science  Anxiety 

When  looking  at  the  data  from  the  science  anxiety  survey  given  before  the  start  and  after  the  conclusion  of 
the  frog  dissection  unit,  the  conventional  treatment  group  showed  significantly  lower  levels  of  anxiety  both  before 
and  after  treatment.  While  at  first  glance  this  appears  puzzling,  closer  inspection  reveals  two  possible  explanations. 

The  first  concerns  the  timing  of  the  unit  of  instruction  and  the  arrangement  of  lab  facilities.  Because  the 
frog  dissection  unit  was  conducted  at  the  end  of  the  academic  year  and  this  was  the  first  time  that  two  alternatives  to 
conventional  dissection  were  to  be  offered,  the  teachers  involved  needed  an  uncommon  level  of  coordination  to  plan 
use  of  lab  facilities.  Unfortunately,  these  teachers  had  no  common  planning  period,  and  therefore  coordination  of 
facilities  had  to  occur  before  and  after  school,  as  well  as  in  the  halls  between  class  periods.  All  these  discussions 
took  place  at  times  when  students  may  have  been  present  and  overheard  the  planning.  This  could  have  resulted  in 
students  having  an  idea  of  which  treatment  their  class  would  receive  and  therefore  may  have  skewed  the  initial 
findings  of  pre-treatment  anxiety  levels. 

A related  explanation  comes  from  the  experiences  of  this  study’s  subject  pool.  The  subjects  within  this 
study  had  had  previous  exposure  to  dissection  activities  during  their  seventh  grade  life-science  curriculum.  While 
these  dissection  activities  were  limited  to  relatively  simple  exercises  in  the  anatomy  of  a worm  and  a squid,  the 
students  were  familiar  with  this  type  of  instructional  activity.  Conversely,  the  students  had  received  no  prior 
exposure  to  a unit  of  instruction  delivered  entirely  through  a desktop  microworld  or  a CD-tutorial.  Koballa  (1993) 
stated  that  activity-oriented  science  instruction  can  help  develop  favorable  attitudes  toward  science.  Previous 
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exposure  to  a like  activity  could  have  skewed  the  results  of  the  science  anxiety  measurements  in  favor  of  the  physical 
dissection  activity. 

When  looking  at  the  level  of  change  in  anxiety  across  treatment  groups,  no  single  treatment  group  showed  a 
significantly  different  change  in  anxiety  compared  to  the  others.  Apparently  the  treatment  received  had  no  bearing 
on  change  in  the  level  of  science  anxiety  exhibited. 

It  is  interesting  to  note  the  relatively  low  levels  of  anxiety  reported  by  students  across  the  board.  All  but 
one  of  the  mean  anxiety  level  scores  fell  below  2.00  on  a range  of  possible  scores  from  1 (Not  at  all  Nervous) 
through  5 (Very  Nervous).  This  finding  may  be  a result  of  the  timing  of  the  study  at  the  end  of  a yearlong,  seventh- 
grade,  life-science  curriculum  that  stressed  active  participation  by  students.  This  follows  the  conclusions  of  Hill  et 
al.  (1995)  that  overall  attitude  toward  science  improved  significantly  when  students  were  engaged  in  active 
participation  within  a seventh  grade  life-science  curriculum  over  time.  This  study  falls  within  their  call  for  additional 
research  to  focus  on  particular  connections  among  specific  science  instructional  activities,  in  this  case,  approaches  to 
a frog  dissection  unit. 

Examining  science  anxiety  measures  by  gender  revealed  some  interesting  patterns.  Females  reported  sig- 
nificantly higher  initial  science  anxiety  than  males.  After  treatment,  although  anxiety  levels  declined  for  both 
genders,  females  continued  to  report  significantly  higher  science  anxiety  than  males  overall. 

The  three  treatment  groups  displayed  varying  results  in  anxiety  levels  by  gender.  The  initial  gender 
difference  within  the  CD-tutorial  group  was  no  longer  present  after  treatment,  suggesting  a beneficial  effect  for  that 
treatment  in  reducing  female  science  anxiety.  For  subjects  in  the  microworld  treatment,  the  difference  was  not 
significant  either  pre-treatment  or  post- treatment.  However,  after  treatment,  the  difference  in  means  for  males  and 
females  did  approach  significance,  with  anxiety  declining  more  among  males  than  females.  It  appears  that  the 
microworld  treatment  was  more  beneficial  for  males.  Finally,  for  the  conventional  dissection  group,  the  difference 
between  genders  was  significant  both  before  and  after  treatment.  The  experience  of  conventional  dissection  had  no 
effect  on  the  disparity  in  science  anxiety  between  males  and  females. 

It  is  interesting  to  note  that  the  CD-tutorial  treatment  seemed  to  reduce  the  level  of  science  anxiety  more  for 
female  students,  while  just  the  opposite  was  occurring  within  the  microworld  treatment  group  as  the  level  of  anxiety 
lowered  more  for  males  after  treatment.  This  might  be  due  to  the  fact  that  students  in  the  microworld  treatment  were 
active  participants  in  their  use  of  the  computer  and  had  much  more  control  over  the  flow  of  information  presented 
than  those  students  in  the  CD-tutorial  treatment,  who  merely  observed  a dissection  taking  place.  Because  the 
microworld  group  actually  performed  a frog  dissection  utilizing  representations  of  dissection  tools,  they  faced  the 
very  real  possibility  of  making  a mistake  in  the  dissection  process.  This  possibility  did  not  exist  for  students  in  the 
CD-tutorial  treatment  as  they  merely  watched  as  the  computer  performed  the  various  dissection  activities. 

Additional  research  is  needed  to  pursue  the  interaction  effect  of  active  versus  passive  role  and  gender  as  to  levels  of 
anxiety  exhibited. 

Overall,  this  study  demonstrated  that  science  anxiety  is  clearly  related  to  gender,  which  confirms  the 
findings  of  Wynstra  (Wynstra,  1991;  Wynstra  & Cummings,  1993).  Wynstra  suggested  that  in  classes  that  involve 
activities  that  could  make  one  squeamish,  alternative  delivery  systems  could  be  used.  In  this  way  students  would 
have  the  option  of  working  with  charts,  models,  or  interactive  computer  software  instead  of  having  to  dissect  once- 
living  creatures  such  as  frogs  or  pigs.  The  results  of  this  study  support  Wynstra’ s findings. 

Our  world  has  rapidly  become  one  based  on  scientific  inquiry  that  crosses  all  economic,  political,  and 
geographical  boundaries.  Educators  have  long  called  for  a fuller,  stronger,  and  more  fundamentally  sound  science 
curriculum  within  our  schools  and  the  development  of  ways  to  encourage  students  to  pursue  a scientific  career  path. 
Atwater,  Wiggins,  and  Gardner  (1995)  concluded  that  the  less  anxious  a student  was  toward  science,  the  more 
positive  that  student  was  toward  the  sciences  in  general.  In  their  review  of  the  research,  Koballa,  et  al.  (1990)  found 
that  the  years  8 to  13  are  the  critical  ages  for  influencing  science  attitudes.  By  understanding  what  components  of  a 
science  unit  of  study  produce  anxiety  amongst  students,  middle-school  teachers  can  better  offset  those  levels  of 
anxiety  with  the  methods  they  use  to  teach  science  concepts. 

This  study  showed  the  desktop  microworld  to  be  beneficial  in  reducing  the  level  of  anxiety  among  males 
while  the  CD-tutorial  helped  in  reducing  female  science  anxiety.  Considering  these  results,  it  can  be  said  that 
technology-based  alternatives  to  conventional  frog  dissection  appear  to  offer  benefits  in  addressing  the  science 
anxiety  issue. 


Preferred  Learning  Style  and  Achievement 

The  data  from  this  study  did  not  support  any  claims  of  a general  effect  on  achievement  for  the  students’ 
preferred  learning  style.  This  was  true  when  looking  at  differences  in  achievement  for  pretest  scores,  posttest  scores, 
and  delayed  posttest  scores.  Within  this  study,  learning  style  alone  does  not  appear  to  be  related  to  achievement. 

It  was  only  when  looking  at  differences  in  gain  scores  by  preferred  learning  style  within  treatment  that 
statistical  differences  began  to  emerge.  The  conventional  dissection  treatment  seems  to  have  the  least  effect  on  those 
students  with  a preference  for  the  visual-kinesthetic  learning  style.  These  students  significantly  under-performed 

290 


275 


their  counterparts  within  the  same  treatment  who  displayed  a preference  for  the  auditory,  the  auditory-kinesthetic,  the 
kinesthetic,  and  the  visual  learning  style  modalities. 

The  micro  world  treatment  seems  to  have  particular  merit  for  students  who  prefer  any  of  the  auditory 
learning  style  modalities  (auditory,  auditory-kinesthetic,  and  auditory- visual)  as  they  outperformed  their  counterparts 
with  a kinesthetic  preference  from  the  pretest  to  the  posttest.  In  fact,  students  with  a preference  for  the  auditory- 
visual  learning  style  had  the  best  performance  overall  of  any  subgroup  in  any  treatment.  They  also  showed 
significantly  higher  gain  scores  than  students  within  the  microworld  treatment  group  who  preferred  the  visual 
learning  style. 

Both  the  conventional  dissection  treatment  and  the  desktop  microworld  alternative  offered  not  only  visual 
and  auditory  channels  but  also  kinesthetic  channels  in  the  form  of  actually  participating  in  the  dissection  process. 
However,  within  the  CD-tutorial  treatment,  students  merely  observed  the  dissection  take  place.  The  findings  of  this 
study  support  the  contention  of  researchers  (Craik  & Tulving,  1975;  Ellis  & Hunt,  1983;  Korwin  & Jones,  1990)  that 
hands-on  activities  help  to  encode  knowledge  acquired  into  long-term  memory  and  that  knowledge  decrement 
(Grieve,  1992)  is  less  following  activities  that  address  all  the  varying  modalities  of  learning.  However,  these  hands- 
on  activities  must  not  be  exclusively  kinesthetic,  but  rather  incorporate  all  modalities  of  learning. 

Limitations  Evident  Within  the  Study 

Several  limitations  were  evident  within  this  study.  They  are  presented  in  no  particular  order,  but  should  be 
addressed  to  more  fully  understand  the  results  of  the  data  obtained. 

To  assess  the  preferred  learning  style  of  a student,  this  study  was  limited  to  data  from  the  school’s  study 
skills  teacher.  The  instrument  used  to  obtain  these  data  allowed  for  categorizing  students  into  groups  reflecting  a 
balance  between  the  three  major  learning  styles:  Auditory,  Visual  and  Kinesthetic.  This  resulted  in  small  numbers 
of  subjects  falling  into  the  shared  categories  of  auditory-kinesthetic  (12),  auditory- visual  (30),  and  visual-kinesthetic 
(21).  Taken  by  themselves  the  numbers  reflected  in  these  groupings  would  not  pose  a problem,  however  when  they 
were  further  divided  by  the  instructional  treatment  received,  the  numbers  in  some  cases  were  very  small.  This 
occurred  most  noticeably  within  the  following  categories:  CD-tutorial  treatment  and  auditory- kinesthetic  (2 
subjects);  CD-tutorial  treatment  and  visual-kinesthetic  (6  subjects);  conventional  treatment  and  auditory- kinesthetic 
(4  subjects);  conventional  treatment  and  visual-kinesthetic  (5  subjects);  and  microworld  treatment  and  auditory- 
kinesthetic  (6  subjects).  Researchers  attempting  similar  future  studies  should  either  utilize  an  instrument  for 
assessing  preferred  learning  style  that  would  not  allow  for  cross-groupings  or  increase  the  number  of  subjects 
included  within  the  pool  to  offset  the  small  numbers  that  might  occur  in  any  one  group. 

This  study  was  conducted  at  the  very  end  of  the  school  year.  Problems  inherent  with  this  timing  are  the 
built-in  distractions  to  students  of  end  of-the-year  activities,  anticipation  of  summer  vacation  looming  in  the 
immediate  future,  and  attrition  due  to  students  leaving  school  early  for  family  vacations  and/or  relocation.  All  these 
factors  can  play  a role  in  the  level  of  concentration  attainable  by  students  and  a reduction  in  the  number  of  subjects 
included  in  the  final  assessment  pool. 

As  stated  previously,  the  timing  of  the  study  left  little  opportunity  for  teachers  to  coordinate  the  use  of 
facilities.  This  led  to  a possibility  of  students  overhearing  teachers  discussing  arrangements  during  periods  of  the 
day  when  students  were  present.  If  this  study  were  to  be  replicated,  care  should  be  given  in  the  discussion  of 
implementation  plans  to  ensure  that  students  are  unaware  of  their  treatment  group  prior  to  the  actual  start  of  the 
instructional  unit. 

Another  limitation  in  this  study  was  the  makeup  of  the  subject  pool.  The  subjects  participating  in  this  study 
were  predominantly  Caucasian  middle-class  suburbanites.  Only  2.1%  fell  within  the  low-income  economic  bracket. 
The  racial  makeup  of  the  population  consisted  of  88%  White,  7.6%  Asian/Pacific  Islander,  1.7%  Black,  2.4% 
Hispanic,  and  0.3%  Native  American.  In  addition,  students  classified  as  Limited-English-Proficient  comprised  only 
2.3%  of  the  school  population.  Taking  all  this  into  consideration,  the  results  of  this  study  cannot  be  extrapolated  to 
the  general  populace,  only  to  those  populations  with  a similar  makeup. 

Suggestions  for  Further  Research 

In  the  course  of  conducting  this  study  and  in  the  analysis  of  the  data  collected,  many  questions  arose  that 
provide  a varied  base  for  further  research.  In  order  to  offset  the  effects  of  previous  experience  on  the  levels  of 
science  anxiety  reported,  studies  should  be  conducted  comparing  the  various  instructional  treatments  using  a hands- 
on  activity  earlier  in  the  year.  It  would  be  necessary  to  limit  the  subject  pool  to  those  students  who  had  no 
experience  in  completing  a scientific  unit  of  study  through  conventional  dissection,  desktop  microworld  or  CD- 
tutorial.  It  would  also  prove  valuable  to  conduct  ethnographic  studies  to  provide  insight  into  particular  connections 
between  science  instruction,  science  attitudes,  science  achievement,  and  cultural  background  for  groupings  such  as 
Asian -American  students  and  African-American  students. 

The  studies  of  Ester  (1994-1995)  and  Messick  (1976)  revealed  a significant  interaction  between 
instructional  approach  and  student  learning  style.  This  study  supports  those  findings  in  the  case  of  the  microworld 
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treatment  but  not  for  the  conventional  treatment  or  the  CD-tutorial  treatment.  The  question  that  naturally  follows  is, 
why  didn’t  the  pattern  that  appeared  within  the  microworld  treatment  also  appear  within  the  conventional  treatment, 
if  both  treatments  incorporated  all  the  modalities  of  learning  within  their  environments?  Further  research  in  this  area 
is  needed  in  order  to  answer  that  question. 

Addressing  the  gender  issue  in  regards  to  the  level  of  science  anxiety  displayed,  several  questions  for  future 
research  emerge.  If  females  exhibit  a higher  level  of  science  anxiety  overall  than  males  during  the  middle-school 
years,  will  that  difference  increase,  decrease,  or  remain  the  same  in  subsequent  years?  And  if  there  is  a change,  what 
are  the  forces  that  cause  that  change?  A follow-up  study  of  the  subjects  within  this  study  as  they  progress  through 
high  school  could  shed  light  on  what  influences  science  anxiety  differences  among  males  and  females. 

The  belief  that  the  effects  of  simulations  are  not  revealed  by  tests  of  knowledge  but  rather  by  tests  of 
transfer  and  application  suggests  further  study.  How  would  the  microworld  treatment  compare  with  the  CD-tutorial 
treatment  if  observations  were  made  on  the  students’  performances  in  later  dissection  procedures  or  life-science 
exercises? 

The  level  of  retention  from  posttest  to  delayed  posttest  bears  a closer  look  in  future  research.  While  most 
tests  within  the  school  curriculum  are  geared  toward  immediate  achievement  gains,  the  goal  of  most  educational 
institutions  is  to  help  students  not  only  learn  but  to  retain  that  which  they  have  achieved.  This  leads  to  the  following 
questions.  Once  knowledge  is  gained,  how  much  is  lost  over  time?  And  which  methods  of  instructional  delivery 
help  the  most  in  the  retention  of  that  knowledge?  Further  research  into  these  areas  is  a must  if  we  are  to  attain  an 
educated  populace. 

There  is  a definite  need  for  more  research  in  the  area  of  desktop  microworlds.  With  access  to  the  World- 
Wide  Web  increasing  throughout  our  schools,  spurred  on  by  government  initiatives,  questions  arise  as  to  what  can  be 
done  with  all  the  information  so  readily  available  once  it  is  accessed.  This  is  particularly  true  for  multimedia 
elements  such  as  digitized  pictures,  sound  and  movies.  Desktop  microworlds  composed  of  these  downloaded 
elements  and  run  by  shareware  programs  offer  a relevant  and  easily  upgradable  instructional  environment.  There  are 
then  many  questions  concerning  these  desktop  microworlds  and  their  role  in  the  instructional  process.  How  best  can 
they  be  incorporated  into  today’s  learning  environments?  Will  they  be  effective  vehicles  for  information  transfer  for 
all  students?  Are  they  more  effective  if  created  by  students  for  students?  Do  they  allow  for  student  control  over  the 
pacing  of  information  flow  and  the  pathways  followed?  If  so,  do  they  allow  for  the  molding  of  the  program  to  the 
student’s  preferred  learning  style?  Just  how  do  teachers  and  students  interact  with  a microworld  to  gain  the  greatest 
benefit? 
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This  paper  proposes  a context -driven  model  of  instructional  design  incorporating  constructivist-oriented 
methods  and  strategies  for  structuring  meaningful , purposeful  learning  environments  into  the  framework  of 
a tradition  systematic  approach.  An  evaluation  study  is  also  presented  which  was  designed  to  investigate 
the  effects  of  implementing  the  context-driven  approach  into  the  initial  planning  of  an  instructional 
program.  Twenty-one  practicing  teachers  were  presented  with  case-based  material  that  elicited  plans  for 
developing  instruction  designed  to  support  the  learning  of  specific  performance  objectives  within  an 
electronic  media- supported  environment.  Each  teacher  developed  two  plans  for  specific  objectives  using 
both  a traditional  systems  approach  as  well  as  the  context-driven  design  model  proposed.  These  plans 
were  analyzed  and  reported  according  to  the  number  and  type  of  instructional  components  and  elements 
included , the  type  of  contexts  described  and  the  role  electronic  media  played  within  these  contexts , the  type 
of  social  interaction  environments  defined  by  the  electronic  media , and  general  attitudes  about  developing 
instruction  according  to  the  two  different  approaches. 

Major  planning  differences  with  respect  to  the  inclusion  of  specific  instructional  components  and  elements 
were  identified  between  the  two  approaches,  as  was  the  role  electronic  media  played  between  the  teachers*  different 
plans.  Differences  between  context  selections  were  also  identified  and  reported.  Implications  for  future  research  are 
included  in  the  paper  as  well. 

Introduction 

Perhaps  the  most  useful  thing  to  emerge  from  Instructional  Technology’s  focus  on  constructivism  during  the 
past  ten  years  has  been  the  critical  examination  of  the  role  context  plays  within  the  instructional  design  process. 
Although  debates  surrounding  the  effects  of  context  within  the  actual  act  of  learning  will  most  likely  continue  for 
quite  a long  time,  the  notion  that  context  could  and  should  play  a more  important  role  in  the  design  and  development 
of  instruction  seems  to  be  more  universally-accepted  among  those  Instructional  Technologists  involved  in  the 
research  and  development  of  improved  systematic  instructional  design  models.  Improving  existing  systematic 
instructional  design  models  through  the  incorporation  of  successfully  tested  and  implemented  constructivist-oriented 
methods  and  strategies  represents  one  of  the  more  important  challenges  facing  Instructional  Technology  today.  The 
purpose  of  this  paper  is  to  propose  one  such  context-driven  model  of  instructional  design  that  merges  methods  and 
strategies  for  structuring  meaningful,  purposeful  learning  environments  into  the  framework  of  a tradition  systematic 
approach.  In  addition,  this  paper  also  presents  the  results  of  a study  designed  to  investigate  the  effects  of 
implementing  a context-driven  approach  into  the  initial  planning  of  an  instruction  program  to  support  the  learning  of 
a specific  performance  objective  within  an  electronic  media-supported  environment. 

Throughout  the  last  decade  a number  of  instructional  design  theories  and  models  have  been  researched  and 
evaluated  which  reflect  a decidedly-constructivist  perspective.  Although  it  would  be  difficult  to  pin  down  a common 
design  methodology  between  these  models,  they  all  seem  to  possess  a unifying  theme:  context.  Models  such  as 
Anchored  Instruction,  Situated  Learning,  and  Constructionism  have  been  widely  evaluated  and  reported,  and  a closer 
examination  of  their  instructional  design  methodologies  reveals  that  they  all  value  the  cultivation  of  rich  learning 
contexts  over  simply  addressing  prescriptive  instructional  goals. 

Anchored  Instruction  represents  a design  model  in  which  learning  a related  set  of  specific  content-area 
skills,  knowledge,  and  attitudes  is  “anchored”  in  a larger  community  or  social  context  (see  Cognition  and 
Technology  Group  at  Vanderbilt,  1992).  Such  anchors  often  include  case-study  or  problems-based  situations,  and 
the  instructional  materials  afford  the  learners  many  opportunities  for  individual  exploration.  Situated  Learning 
represents  a similar  approach  to  the  structuring  of  instructional  environments.  Within  this  model,  instruction 
directed  toward  the  learning  of  individual  performance  objectives  is  situated  within  a context  reflecting  a culture  in 
which  acquiring  and  practicing  the  identified  skills  “naturally”  occurs  (see  Lave  & Wenger,  1991).  Social 
interaction  is  also  a critical  component  of  Situated  Learning  environments,  as  the  learning  culture  itself  is  primarily 
defined  by  the  interpersonal  interactions  of  its  members. 

Constructionism  represents  a subtle  departure  from  other  constructivist  learning  strategies.  This  model, 
attributed  to  Seymour  Papert  (1993),  places  a special  importance  on  the  role  of  actual  constructions  in  the  learning 
environment  as  a support  for  knowledge  construction  in  the  head.  Within  this  theoretical  framework,  effective 
learning  is  both  facilitated  and  represented  by  the  action  of  individual  learners  immersed  in  creation. 

Each  of  these  “constructivist-friendly”  instructional  models  could  certainly  be  viewed  as  providing 
opportunities  for  the  learners  to  construct  their  own  meaning  to  instructional  stimuli  and,  consequently,  direct  any 
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number  of  instructional  events.  Attempting  to  represent  these  models  within  more  traditional  instructional  systems 
design  frameworks  could  take  many  different  directions.  For  example,  each  of  these  theories/models  might  be 
viewed  as  simply  various  ways  to  ensure  that  the  learning  of  individual  SKA  occur  within  contexts  that  are 
meaningful  and  purposeful  to  learners.  They  could  also  be  viewed  as  nothing  more,  or  less,  than  very  complex 
motivational  strategies.  Although  most  “traditional”  systematic  instructional  design  models  are  grounded  in 
behavioralist  ideology,  the  overall  purpose  of  any  ISD  model  is  to  maximize  the  probability  that  learners 
experiencing  instruction  will,  in  fact,  learn  what  the  designers  intend.  Regardless  of  whether  or  not  an  instructional 
designer  considers  herself  or  himself  a behavioralist  or  a constructivist  (or  an  agnostic  for  that  matter),  most  seem  to 
accept  the  idea  that  learners  will  construct  their  own  meaning  to  any  information  they  receive  through  interacting 
with  their  environment.  Learner  interpretation  of  external  stimuli  will  be  influenced  by  their  previous  experiences  as 
well  as  their  cognitive  abilities,  any  social  negotiations  presented  within  the  learning  environment,  the  type  of 
instructional  strategies  presented,  the  general  manner  in  which  messages  are  presented,  overall  motivational  levels, 
and  any  other  physiological  and/or  emotional  factors  which  define  how  individuals  personally  want  to  learn  things. 
Very  few  instructional  designers  would  argue  that  learners  interpret  events  and  ascribe  their  own  meaning  to  external 
stimuli,  generally  speaking,  this  constitutes  one  of  the  fundamental  design  problems  facing  instructional  designers. 

It  is  their  primary  task  to  structure  learning  environments  which  maximize  the  probability  that  all  (or  most)  of  the 
learners  will  interpret  events  and  assign  or  construct  common  meanings  enabling  them  to  learn  the  prescribed, 
defined  SKA.  In  fact,  if  learners  didn’t  “construct”  their  own  meanings  to  events  then  ID  would  probably  be  a much 
simpler  affair. 

A Context-Driven  Model 

Many  “traditional”  instructional  design  models  include  fundamental  steps  similar  to  those  detailed  in  Dick 
and  Carey’s  (1996)  systematic  model  of  instructional  design: 

1 Assess  needs  to  identify  goal(s) 

2.  Conduct  instructional  analysis  (including  learner  and  context  analysis) 

3.  Write  performance  objectives 

4.  Develop  assessment  instruments 

5.  Develop  instructional  strategy  (conditions  dependent  on  type  of  skill  to  be  learned) 

6.  Develop  and  select  instructional  material 

7.  Design  and  conduct  formative  evaluation  of  instruction 

8.  Revise  instruction  (steps  2-7) 

9.  Design  and  conduct  summative  evaluation 

The  individual  components  of  this  model  are  relatively  consistent  with  the  primary  steps  described  in  other 
popular  instructional  design  models,  including  Reiser  and  Dick  (1996);  Sullivan  and  Higgins  (1983),  Hunter  (1982), 
and  Merrill,  Li,  & Jones  (1990).  Instructional  components  aren’t  the  only  similarity  between  these  different  models. 
Another  basic  similarity  between  these  different  instructional  design  models  is  reflected  in  their  treatment  of 
instructional  strategy  development.  All  employ  some  type  of  conditions -based  approach,  meaning  that  the  most 
appropriate  type  of  instructional  strategy  for  any  given  design  is  dependent  on  the  type  of  skill  to  be  facilitated. 

Although  any  given  conditions-based  instructional  design  prescription  may  incorporate  a number  of 
motivational  strategies,  none  of  the  more  popular  instructional  design  models  address  the  role(s)  that  the  entire 
learning  context  plays  within  every  aspect  of  the  instructional  decision-making  and  implementation  process.  It  could 
be  argued  that  the  deconstruction  of  instructional  events  for  the  purpose  of  designing  instruction  for  given 
performances  can  effectively  occur  outside  specific  contexts.  But  if  developing  contextual  learning  environments  is 
the  goal  of  the  designer,  then  the  identification  of  a meaningful,  purposeful  context  as  well  as  the  role(s)  that  the 
media  used  to  define  the  context  will  play  within  the  entire  instructional  experience  should  be  incorporated  into  the 
instruction  design  process  from  the  beginning.  Since  many  instructional  design  models  today  are  directed  toward  the 
use  of  some  form  of  electronic  media,  it  makes  sense  to  take  advantage  of  the  manner  in  which  electronic  media  are 
commonly  defined  with  respect  to  the  context  they  help  facilitate  (see  Figure  4).  By  capitalizing  on  the  emerging 
roles  that  electronic  media  are  playing  within  the  real  world,  instructional  designers  can  begin  to  construct  more 
well-defined  context-driven  instructional  models  which  designed  to  facilitate  more  meaningful,  purposeful 
technology-supported  learning  environments.  The  following  illustrates  an  instructional  design  model  that  is  driven  by 
a context  integrating  electronic  media  into  the  instructional  planning  and  implementation  process: 
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1 . Assess  needs  and/or  identify  instructional  goal(s) 

2.  Identify  a context  in  which  goal  is  applied/practiced  in  the  real  (or  the  learner’s)  world 

3.  Decide  which  electronic  media-supported  context(s)  best  approximate  the  essence  of  the  real  world  context  (see  Appendix 
E): 

4.  Revise  instructional  goal  to  reflect  identified  learning  context 

5.  Conduct  instructional  analysis  and  write  objectives  related  to  both  goal  (content)  AND  context 

6.  Develop  assessment  instruments 

7.  Conceptualize  and  articulate  a “Big  Picture”  for  the  planned  instructional  program  based  on  the  identified  prerequisite 
skills,  the  context  to  be  established,  the  content  area,  and  the  instructional  goal 

8.  Develop  instructional  strategy  (see  Figure  1 ) in  which  conditions  (see  Figure  3)  are  dependent  on  type  of  skill  to  be 
leamed(  see  Figure  2) 

9.  Develop  and  select  instructional  material 

10.  Design  and  conduct  formative  evaluation  of  instruction 

11.  Implement  instruction 

1 2.  Revise  instruction  (Steps  2- 1 0) 

Like  its  predecessors,  this  context-driven  model  of  electronic  media-supported  instructional  design  is  also 
conditions-based.  Unlike  other  design  models,  however,  it  emphasizes  the  identification  of  a meaningful  learning 
context  very  early  in  the  design  phase,  and  the  context  itself  is  incorporated  into  the  goal  and  subsequent 
instructional  analysis.  Context  identification  is  based  on  two  important  elements: 

• How  the  instructional  goal  “practiced”  in  the  real  world 

• How  electronic  media  help  approximate  real-world  contexts 


Although  it  would  impossible  to  systematize  the  identification  of  how  any  given  goal  might  be  practiced  in 
reality,  it  is  not  quite  so  difficult  to  categorize  the  manner  in  which  electronic  media  have  traditionally  established 
and  cultivated  different  types  of  contexts.  Many  books  dealing  with  electronic  media  and  education  have  identified 
contexts  similar  to  those  described  in  Figure  4:  Creation/Construction,  Situation  Exploration  (including  case-based 
situations),  Simulation,  Reference  Exploration,  Tutorial  (direct  instruction),  Drill  and  Practice,  Game  (cooperative, 
competitive,  puzzle),  Communication,  “Real”  (Allesi  &Trollip,  1991;  Merrill,  et  al,  1996;  Grabe  & Grabe,  1996). 
Since  these  context  categories  represent  time-tested  development  enterprises  which  have  conformed  to  the  various 
parameters  of  electronic  media  itself  (specifically  computers),  it  makes  sense  to  use  them  as  guidelines  for 
instructional  designers  charged  with  the  task  of  developing  instruction  specifically  for  a particular  electronic 
medium.  Although  a strong  argument  could  be  made  against  the  development  of  an  instructional  design  model  that 
incorporates  media-specific  strategies  and  procedures,  in  today’s  electronic  media-rich  work  and  school  cultures  it  is 
common  practice  to  articulate  instructional  needs  and  media  needs  in  the  same  statement. 

As  indicated  earlier,  this  context-driven  model  follows  a traditional  systematic  instructional  design 
framework  while  incorporating  the  learning  context  into  most  of  the  decision-making  processes.  Once  an 
instructional  goal  is  clearly  identified,  the  designers  are  directed  to  identify  a meaningful  context  based  on  how  the 
goal  is  practiced  in  the  learners’  real  world.  The  instructional  designers  are  then  presented  with  the  listing  of 
electronic  media  contexts  (Figure  4)  and  encouraged  to  categorize  their  chosen  context  into  one  or  more  of  the 
contrived  types.  This  step  may  also  aid  in  the  selection  of  a meaningful  learning  context  if  the  skills  indicated  within 
the  goals  aren’t  readily  identifiable  as  real-world  performances  (see  most  Math  and  Social  Studies  curricula  for 
plenty  of  examples). 

After  a meaningful  learning  context  has  been  identified,  the  instructional  designer  is  directed  to  review  the 
initial  instructional  goal  and  incorporate  the  context  into  the  goal  if  the  context  will  require  the  learning  of  context- 
specific  outcomes.  For  example,  consider  the  following  goal  statement:  “Given  the  length  of  one  side  of  a right 
triangle  as  well  as  the  internal  angle  measurements,  the  learners  will  use  the  Pythagorean  Theorem  to  calculate  the 
lengths  of  the  unknown  sides.”  Since  the  Pythagorean  Theorem  is  often  used  to  calculate  the  height  of  objects  too 
tall  to  measure  directly,  the  designer  may  choose  a context  in  which  the  learners  must  calculate  the  height  traveled  by 
a model  rocket.  The  revised  goal  statement  might  then  read  “Given  the  distance  from  an  observer  and  a rocket 
launch  pad  (meters)  along  with  the  length  (meters)  of  the  hypotenuse  for  the  right  angle  formed  between  the 
observer,  the  launch  pad,  and  the  rocket  at  its  zenith,  the  students  will  calculate  the  height  of  the  rocket  to  the  nearest 
meter.”  This  new  goal  statement  focuses  the  entire  instructional  design  experience  around  rockets  and  how  math  can 
be  used  to  make  calculated  inferences  about  observed  events.  And  including  the  context  within  the  goal  statement 
itself  helps  ensure  that  context-dependent  skills  (such  as  “define  zenith”)  are  addressed  in  the  subsequent  design. 

The  next  step  in  the  design  process  requires  performing  a complete  instructional  analysis  of  the  goal/context 
to  identify  all  subordinate  skills  that  must  be  learned  in  order  to  accomplish  the  stated  goal  within  the  identified 
context.  Following  the  instructional  analysis,  all  performance  outcomes  are  sequenced  and  placed  into  units  or 
lessons.  Each  lesson  is  then  treated  separately,  with  each  lesson  developed  according  to  the  instructional  design 
components  indicated  in  Figure  1. 
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Experience  with  context-driven  instructional  design  has  indicated  that  certain  context  types  lend  themselves 
better  than  others  to  the  implementation  of  specific  instructional  design  components.  Tessmer  and  Richey  (1997) 
defined  the  role  of  any  context  as  orienting,  instructional,  and/or  transfer.  These  categories  of  context  “functions” 
are  easily  correlated  to  those  instructional  design  components  included  in  Figure  1.  This  correlation  indicates  that  an 
orienting  context  can  successfully  accomplish  some  or  all  of  an  effective  instructional  program  introduction,  an 
instructional  context  can  accommodate  part  of  an  introduction,  the  activities/information,  the  practice/feedback,  and 
perhaps  the  review.  And  a transfer  context  can  accommodate  those  elements  within  an  effective  transfer  component. 
Classifying  the  type  of  context  and  then  determining  its  function  is  useful  in  identifying  those  components  of  good 
instructional  design  that  may  not  be  fully  accommodated  by  the  chosen  context.  If  deficiencies  in  the  design  are 
detected,  the  designer  is  encouraged  to  develop  additional,  specific  instructional  components  that  complement  the 
context,  but  are  separate  from  it  (A  good  example  of  this  occurred  during  the  development  of  the  Jasper  Woodbury 
Problem-Solving  material.  See  CTGV,  1992). 

By  following  the  basic  instructional  design  steps  indicated,  the  instructional  designer  is  encouraged  to 
create  an  effective  learning  experience  that  is  integrated  within  a purposeful,  meaningful  environment.  Since  the 
contrived  contexts  described  in  this  model  (Figure  4)  were  derived  from  a review  of  the  way  computer-based 
environments  are  typically  described,  perhaps  a model  such  as  the  context-driven  model  described  in  this  paper  will 
further  the  cultivation  and  definition  of  those  attributes  of  electronic  media  that  can  provide  meaning  and  purpose  to 
learning. 

As  this  context-driven  model  of  instructional  design  was  being  developed  and  tested,  a number  of  research 
and  evaluation  experiences  took  place.  On  one  specific  occasion,  the  model  was  evaluated  formally  against  a more 
traditional  systems  approach  in  the  initial  planning  stages  of  design  by  teachers  enrolled  in  an  advanced  instructional 
design  and  evaluation  course.  During  this  evaluation  study,  identical  procedures  and  treatment  materials  were 
implemented,  and  these  tests  will  be  referred  to  in  the  following  section  as  ‘The  Evaluation  Study.” 

The  Evaluation  Study:  Introduction 

The  purpose  of  this  evaluation  study  was  to  explore  the  effects  of  using  a context-driven  model  of 
instructional  design  on  the  initial  planning  of  electronic  media-supported  instruction.  Practicing  K-12  teachers  were 
charged  with  the  task  of  planning  instruction  for  specific  instructional  objectives  while  incorporating  the  use  of 
specific  electronic  media,  and  they  were  compelled  to  articulate  their  instructional  plans  through  both  a traditional 
instructional  design  approach  and  the  context-driven  method  described  earlier  in  this  paper.  This  investigation  was 
conducted  in  an  effort  to  answer  the  following  questions  about  possible  differences  in  the  instructional  plans 
developed  by  novice  designers  using  a more  traditional  systems  approach  versus  a context-driven  design: 

• Will  practicing  teachers  incorporate  different  numbers  and  types  of  effective  instructional  components/elements  between 
their  two  plans? 

• Will  they  make  different  decisions  about  the  role  electronic  media  should  play  in  the  learning  environment  between  their 
two  plans? 

• Will  the  manner  in  which  student  interactions  with  each  other  and  the  media  be  different  between  the  two  plans? 

• Will  practicing  teachers’  attitudes  about  designing  instruction  be  different  between  the  two  plans? 

Previous  to  this  evaluation  study,  it  had  been  the  experience  of  the  author  that  a context-driven  approach  to 
instructional  design  not  only  helped  to  produce  more  effective  instructional  programs  than  more  traditional 
approaches,  but  that  utilizing  a context-driven  approach  made  it  easier  for  instructional  design  novices  to  describe 
instructional  plans  that  inherently  included  effective  design  components.  It  had  also  been  demonstrated  informally 
that  forcing  designers  to  categorize  the  role  electronic  media  would  play  in  the  design  process  into  one  of  the 
contrived  categories  described  in  Figure  4 resulted  in  the  media  itself  playing  a bigger  part  in  the  planned 
instructional  program.  In  addition,  the  task  of  developing  instruction  seemed  to  be  more  enjoyable  by  those  novice 
developers  assigned  to  projects  involving  the  comprehensive  design  and  development  of  electronic  media-supported 
instruction.  These  observations  prompted  the  author  to  take  a more  empirical  approach  toward  identifying  the  value 
of  a context-driven  instructional  design  model. 

The  Evaluation  Study:  Method 
Subjects 

The  twenty-one  subjects  for  this  evaluation  study  were  all  practicing  teachers  enrolled  in  either  an 
Instructional  Technology  (IT)  or  a Curriculum  and  Instruction  (Cl)  graduate  program  at  a major  research  and 
teaching  university  in  the  eastern  United  States.  These  students  were  enrolled  an  instructional  design  and  evaluation 
course  designed  to  teach  an  instructional  systems  design  (ISD)  model  similar  to  Dick  and  Carey’s  (1996)  general 
systems  design  framework  described  earlier  in  this  paper. 


Procedures 

The  treatments  for  this  evaluation  study  consisted  of  a case-based  instructional  design  scenario  which 
elicited  two  different  instructional  plans  following  both  a “traditional”  instructional  systems  as  well  as  a context- 
driven  planning  design  approach.  The  case  prompted  the  subjects  to  “...imagine  you  are  an  instructional  design 
consultant  currently  working  for  a publishing  company  interested  in  developing  instructional  material  to  support 
social-studies  Standards  of  Learning  (SOL)  for  K-8  schools.  This  company  has  previously  developed  a variety  of 
electronic  media-supported  material  addressing  math  and  English  SOL’s  and  successfully  marketed  them  to  a 
number  of  school  districts.  Electronic  media,  in  this  case,  refers  to  the  use  of  computers,  video,  and  audio  within  the 
instructional  program,  and  most  of  the  school  administrators  and  teachers  indicated  that  the  electronic -media  aspects 
of  the  material  are  what  they  really  valued  about  the  instruction.  The  role  electronic  media  played  within  these 
successfully- implemented  instructional  programs  ranged  from  simply  providing  information  and  examples  in 
conjunction  with  traditional  text-based  material  to  complete,  individualized  direct-instruction  multimedia  computer 
programs  presenting  information,  practice  and  assessment.  You  have  been  enlisted  by  the  company  to  create  a plan 
for  developing  electronic  media- supported  instruction  for  one  specific  7th-grade  social  studies  standard.  No 
decisions  have  been  made  about  the  type  of  mediated  material  to  be  developed  or  the  role  that  the  media  itself  will 
play  in  the  finished  instructional  product,  but  all  the  electronic  media-based  components  of  each  company  products 
are  designed  to  accommodate  the  following  general  classroom/school  technology  parameters:  25  students  per  class, 
three  networked  (Internet  access)  multimedia  computers  in  each  classroom,  one  25-inch  television  connected  to  a 
VCR  as  well  as  one  of  the  classroom  computers,  one  audio  “boom  box”  with  cassette  tape  and  CD  player,  one 
overhead  transparency  projector,  and  access  to  a school  computer  lab  with  16  networked  multimedia  computers  for  a 
minimum  of  2 hours  per  week.” 

As  indicated,  the  subjects  were  instructed  to  design  a plan  for  developing  electronic  media-supported 
instruction  for  a specific  social  studies  SOL.  The  two  standards  (one  per  type  of  design)  used  within  the  case  were 
taken  from  the  Standards  of  Learning  for  Virginia  Public  Schools  (Virginia  Board  of  Education,  1995): 

The  student  will  compare  the  American  political  and  economic  system  to  systems  of  other  nations,  including  Japan, 
China,  and  leading  Western  European  nations,  in  terms  of 

• governmental  structures  and  powers; 

• the  degree  of  governmental  control  over  the  economy;  and 

• entrepreoneurship,  productivity,  and  standards  of  living. 

7.10  The  student  will  interpret  maps,  tables,  diagrams,  charts,  political  cartoons,  and  basic  indicators  of 
economic  performance  (gross  domestic  product,  consumer  price  index,  productivity,  index  of  leading  economic 
indicators,  etc.)  for  understanding  of  economic  and  political  issues. 

Following  the  portion  of  the  ID  course  presenting  the  basic  ISD  model  but  before  examining  the  specific 
role(s)  context  plays  within  the  design  process,  the  subjects  were  presented  with  the  case-based  scenario  for  the 
current  study.  They  were  informed  that  the  material  (represented  as  a course  “pretest”)  was  going  to  present  them 
with  an  instructional  design  scenario  followed  by  a series  of  questions.  They  were  instructed  to  answer  each 
question  with  as  much  detail  as  needed  to  convey  their  messages  clearly  yet  concisely.  They  were  also  told  that  their 
answers  were  not  going  to  be  graded  for  points;  rather,  all  answers  were  going  to  be  used  as  part  of  the  second 
portion  of  the  course.  In  addition,  the  subjects  were  instructed  to  answer  each  question  without  looking  ahead  to 
subsequent  items  or  going  back  to  change  any  answers  once  they  had  initially  completed  them. 

The  first  treatment  experience  consisted  of  the  case  presentation,  with  19  randomly-assigned  students 
receiving  standard  7.8  (‘The  student  will  compare  the  American  political  and  economic  system...”)  and  the  other  19 
receiving  standard  7.10  (‘The  student  will  interpret  maps,  tables,  diagrams...”)  as  the  specific  objective  for  which 
they  needed  to  plan  electronic  media-supported  instruction.  Subjects  were  then  asked  to  respond  to  the  following 
questions,  corresponding  to  a more  traditional  instructional  systems  planning  approach: 

1.1  List  all  the  non-electronic  and  electronic  media  (textbooks,  hand-outs,  worksheets,  teacher-led  lectures,  video,  audio, 
filmstrip,  computers,  etc.)  that  this  instructional  program  will  employ  and  describe  the  role  each  will  play  within  the 
instructional  program. 

1.2  Describe  how  your  instructional  program  will  introduce  the  selected  topic  to  the  learners. 

1.3  Describe  the  activities  and/or  information  that  the  learners  will  experience  within  the  instructional  program 

1.4  If  applicable,  describe  the  type  of  practice  and  feedback  that  the  learners  will  experience  within  the  instructional  program 

1.5  If  applicable,  describe  the  review  that  the  learners  will  experience  before  the  posttest. 

1.6  If  applicable,  describe  the  posttest  that  the  learners  will  experience  within  this  instructional  program 

1.7  If  applicable,  describe  any  opportunities  in  which  the  program  will  provide  the  learners  with  an  opportunity  to  apply  the 
skills,  knowledge  and  attitude  indicated  within  the  standard  to  another  situation. 

Following  their  experience  with  the  case  scenario  as  it  applied  to  their  first  assigned  standard  (objective), 
the  subjects  were  asked  to  perform  the  same  task  for  the  other  standard.  This  time,  however,  they  were  asked  to 
answer  the  following  set  of  questions  reflecting  a context-driven  planning  approach: 
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2. 1 Describe  a situation  in  which  the  skill(s)  indicated  in  Standard  7.X  are  used  in  the  real  world. 

2.2  Review  the  different  types  of  contexts  that  electronic  mediated  learning  environments  support  (see  Figure  4)  and  select  a 
context  type  that  you  feel  could  be  structured  or  cultivated  within  the  instructional  program  you  are  developing  which 
might  approximate,  in  some  way,  the  real-world  application  described  in  2.1.  State  the  context  type  (from  the  choices 
provided  in  Figure  1)  you  plan  to  develop  within  your  program,  and  briefly  describe  how  it  will  be  structured. 

2.3  Rewrite  the  standard  so  that  it  reflects  the  context  you  have  selected  as  well  as  the  skill  it  originally  communicated. 

2.4  List  all  the  non -electronic  and  electronic  media  (textbooks,  hand-outs,  worksheets,  teacher-led  lectures,  video,  audio, 
Filmstrip,  computers,  etc.)  that  this  instructional  program  will  employ  and  describe  the  role  each  will  play  within  the 
instructional  program. 

2.5  Describe  how  your  instructional  program  will  introduce  the  selected  topic  to  the  learners  within  the  context  you  have 
structured. 

2.6  Describe  the  activities  and/or  information  that  the  learners  will  experience  within  the  within  the  context  you  have 
structured. 

2.7  If  applicable,  describe  the  type  of  practice  and  feedback  that  the  learners  will  experience  within  the  context  you  have 
structured. 

2.8  If  applicable,  describe  the  review  that  the  learners  will  experience  before  the  posttest. 

2.9  If  applicable,  describe  the  posttest  that  the  learners  will  experience  within  this  instructional  program. 

2.10  If  applicable,  describe  any  opportunities  in  which  the  program  will  provide  the  learners  with  an  opportunity  to  apply  the 
skills,  knowledge  and  attitude  indicated  within  the  standard  to  another  situation. 

Upon  completing  the  second  set  of  planning  questions,  the  subjects  were  directed  to  respond  to  a series  of 
items  reflecting  their  attitudes  about  planning  electronic  media-supported  instruction  for  the  two  different  standards. 

Responses  to  each  question  were  evaluated  and  tabulated  by  the  author.  In  an  effort  to  determine  how  well 
each  instructional  plan  corresponded  to  specific  instructional  design  criteria,  the  responses  for  items  1.2-1. 7 as  well 
as  2.S-2.9  were  categorized  according  to  the  instructional  design  components  and  elements  indicated  within  Figure  1. 
For  example,  items  1.2  and  2.5  asked  the  subjects  to  describe  “..how  your  instructional  program  will  introduce  the 
selected  topic  to  the  learners. . .”  Responses  to  this  item  were  evaluated  based  on  how  many  of  the  following 
elements  of  an  effective  introduction  were  included  in  the  answer: 

• Gain  learner  attention 

• Articulate  in  some  way  the  SKA  already  needed  to  succeed  within  the  new  learning  environment 

• Identify  opportunities  in  which  learners  will  relate  what  is  about  to  be  learned  (goal)  to  what  they  already  know  how  to  do 

• Inform  learners  of  objectives  (o.k.  to  be  vague) 

• Fit  objective(s)  into  a "Big  Picture" 

• Present  the  utility  (relevance)  of  the  SKA  to  be  learned 

• Clearly  identify  the  incentives/rewards  for  learning  the  SKA  and  succeeding  within  the  learning  environment 

• Establish  clearly-perceived  learner  accountability,  role(s)  and  task(s)  within  the  learning  environment 

• Establish  clearly-perceived  instructor  role(s)  and  learner  support  mechanisms 

• Employ  specific  content-area  methodologies  if  applicable  (ensuring  that  all  other  elements  are  properly  addressed) 

Similar  evaluation  methods  were  conducted  on  each  item  related  to  the  other  instructional  design 
components.  In  addition  to  the  tabulation  of  instructional  design  element  frequencies,  the  type  of  media  selected  as 
well  as  their  described  functions  were  also  recorded.  A comparison  of  the  different  context  types  by  electronic 
media  selection  was  also  determined  between  those  plans  facilitated  within  the  traditional  model  and  those  plans 
facilitated  within  the  context-driven  approach. 

Results 


As  described  earlier,  individual  responses  to  each  treatment  item  were  categorized  and  then  tabulated 
according  to  each  standard  (7.8  ‘The  student  will  compare  the  American  political  and  economic  system.. .”  versus 
7.10  ‘The  student  will  interpret  maps,  tables,  diagrams. . .”)  and  design  approach  (traditional  versus  context-driven). 
The  purpose  of  initially  tabulating  data  by  standard  was  simply  to  determine  if  any  major  differences  existed  between 
responses  to  the  two  different  standards  themselves  regardless  of  which  design  approach  was  utilized.  Although 
some  minor  differences  were  identified,  it  was  the  judgement  of  the  author  that  a pattern  of  differences  between  the 
two  standards  did  not  exist;  consequently,  no  formal  reporting  or  discussion  will  ensue  with  respect  to  this  variable 
alone. 

Table  1 reports  the  results  of  analyzing  the  treatment  items  that  elicited  specific  responses  to  the  planning  of 
discrete  instructional  design  components.  These  data  indicate  that  a total  of  65  elements  of  an  effective  Introduction 
were  described  within  the  plans  following  a traditional  ID  planning  approach,  and  98  elements  of  an  effective 
introduction  were  described  within  the  plans  following  the  context-driven  approach.  A breakdown  of  individual 
introduction  elements  (delineated  in  Figure  1)  are  not  included  in  Table  1. 

The  data  presented  in  Table  1 also  displays  the  number  of  Activity/Information  design  elements  included  in 
the  planning  responses.  These  data  indicate  that  93  total  elements  were  described  in  the  plans  for  the  subjects’ 
responses  using  the  traditional  deign  approach,  while  103  total  elements  were  described  in  the  plans  for  responses 
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using  the  context-driven  approach.  Table  1 also  reports  the  number  of  Practice/Feedback  elements  included  within 
the  instructional  plans.  These  data  indicate  similar  totals  for  each  type  of  design,  with  58  instances  of 
Practice/Feedback  elements  reported  within  the  plans  facilitated  through  the  traditional  model  and  62  elements  were 
identified  within  the  plans  described  using  the  context-driven  design  approach.  Likewise,  responses  between  design 
approaches  for  the  Review  and  Assessment  components  were  very  similar.  The  data  in  Table  1 indicate  that  35 
review  elements  were  described  within  the  traditional  design  approach  plans  while  31  instances  of  review  elements 
were  detailed  within  the  context-driven  plans.  In  addition,  the  data  in  Table  1 indicate  that  22  assessment  plans 
described  within  the  traditional  approach  explicitly  matched  the  “...performances  and  conditions  indicated  within 
objectives,”  while  only  15  plans  using  the  context-driven  approach  described  such  element.  The  final  instructional 
design  component  analyzed  was  Transfer,  and  there  appeared  to  be  little  difference  in  the  number  of  transfer 
elements  described  within  each  design  approach.  The  data  in  Table  1 indicate  that  15  instances  of  transfer  elements 
were  described  by  subjects  using  the  traditional  approach,  and  15  instances  were  described  within  the  responses 
elicited  from  the  context-driven  approach. 

Table  2 reports  the  data  for  the  classification  of  contexts  described  within  the  instructional  plans,  and  the 
frequency  of  different  context  types  are  broken  down  further  by  type  of  media  utilized.  Although  the  treatment 
materials  specifically  encouraged  the  subjects  to  select  a single  context  within  their  context  driven  plans  from  the 
descriptions  provided  in  Figure  4,  most  plans  (context  and  traditional)  included  descriptions  for  more  than  one 
context  type.  These  data  indicate  that  five  Creation  contexts  were  described  by  subjects  within  those  plans  which 
followed  the  traditional  design  approach,  while  9 were  described  by  subjects  using  the  context-driven  design.  All 
contexts  in  this  case  were  described  as  facilitated  through  the  use  of  computers.  Two  plans  following  the  traditional 
approach  reported  Simulation  contexts  facilitated  with  the  help  of  computers,  and  one  traditional  plan  described  a 
video-supported  Simulation.  This  contrasts  with  1 1 computer-based  Simulation  contexts  described  within  the 
context-driven  plans,  and  one  video-supported  Simulation  context  described  within  a context-driven  plan. 

The  data  in  Table  2 also  indicate  that  ten  traditional  ED  plans  described  Situation  Exploration  contexts,  with 
six  of  these  computer-based  and  four  video-supported.  Sixteen  Situation  Exploration  plans  were  described  within 
the  context-driven  plans,  with  13  being  computer-based  and  three  video-supported.  Also,  14  Reference  Exploration 
contexts  were  described  within  the  traditional  plans  (11  computer-based,  three  video)  while  only  nine  of  these 
contexts  were  described  within  the  context-driven  plans  (seven  computer-based,  two  video).  Seven  traditional  plans 
and  five  context-driven  plans  described  computer-based  tutorial  contexts,  and  14  Drill  and  Practice  contexts  were 
described  within  traditional  plans  (10  computer-based,  one  video,  two  overhead  transparency-based,  and  one  audio) 
while  only  five  computer-based  Drill  and  Practice  environments  were  identified  in  the  context-driven  plans. 

Table  2 also  displays  data  indicating  that  six  traditional  plans  described  Game  contexts  (five  computer- 
based,  one  video)  while  only  three  computer-based  Game  contexts  were  detailed  in  the  context-driven  plans.  In 
addition,  seven  computer-based  Communication  contexts  were  described,  all  within  the  context-driven  plans.  The 
data  in  Table  2 also  indicate  that  one  computer-based  Real  context  was  depicted  within  a traditional  plan,  while  six 
Real  contexts  (five  computer-based,  one  video)  were  identified  in  the  context-driven  plans. 

Table  2 also  reports  data  for  the  Linear  Information  Presentation  context,  representing  those  instructional 
plan  descriptions  which  utilized  media  only  to  present  information  to  the  learners  in  linear  fashion  with  little  or  no 
implied  interaction.  These  contexts  were  only  identified  within  the  traditional  plans,  with  five  computer-based,  two 
video,  one  overhead  transparency,  and  two  audio  contexts  described. 

The  data  in  Table  3 present  information  about  the  general  type  of  social  interaction  environments  facilitated 
by  the  different  types  of  electronic  media  within  the  instructional  plans.  Instances  of  individual,  small  group,  and/or 
entire  class  structures  were  identified  and  categorized  by  electronic  media  type.  These  data  indicate  that  only  one 
traditional  plan  described  computers  facilitating  an  entire  class  experience,  seven  plans  described  small  group  and  14 
plans  described  individual  activities.  Computers  were  described  as  facilitating  two  entire  class  activities  within  the 
context-driven  plan  descriptions,  18  small  group  activities,  and  six  individual  learning  experiences. 

Table  3 also  displays  data  for  the  type  of  social  interaction  environments  defined  within  video- supported 
plans.  These  data  indicate  that  nine  entire  class  experiences  were  defined  within  the  traditional  plans,  and  three 
small  group  as  well  as  two  individual  activities  were  identified.  Video  in  the  context-driven  models  was  used  for 
entire  class  activities  within  three  plans,  five  plans  described  small  group  while  two  described  individual  video- 
supported  learning  activities.  Use  of  the  overhead  projector  was  not  described  in  many  plans,  with  one  class  and  one 
small  group  experience  identified  within  the  traditional  models,  and  only  one  entire  class  experience  depicted  within 
a context-driven  plan.  Similar  results  were  identified  for  the  use  of  audio,  with  two  class  and  one  individual 
experience  described  within  traditional  plans  and  no  audio  experiences  detailed  in  the  context-driven  plans. 

Finally,  responses  to  a few  attitudinal  items  were  tabulated.  The  first  attitude  item  asked  the  subjects  to 
identify  “.  ..which  SOL  was  it  easiest  to  develop  instruction?”  Six  subjects  responded  that  standard  7.8  (‘The 
student  will  compare  the  American  political  and  economic  system. . .)  was  easiest,  1 1 felt  that  standard  7. 10  (“The 
student  will  interpret  maps,  tables,  diagrams,  charts. . .)  was  easiest,  and  four  reported  that  they  were  both  the  same. 
When  asked  “If  each  instructional  program  were  to  be  developed  according  to  your  plan,  which  do  you  think  would 
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be  more  effective  for  the  target  audience  (7th  graders)?”  12  subjects  responded  that  their  context-driven  plans  would 
be  more  effective,  while  only  three  indicated  that  their  traditional  plans  would  be  more  effective.  Six  subjects  felt 
that  they  would  both  be  equally  effective.  Similarly,  when  asked  “If  you  were  given  the  job  of  actually  creating  the 
instructional  program  you  described,  which  would  you  rather  develop?”  15  subjects  responded  that  they  would  rather 
develop  the  instructional  program  following  the  context-driven  plan,  three  responded  that  they  would  rather  develop 
the  program  described  within  their  traditional  plan,  and  three  responded  that  it  wouldn't  make  any  difference  to 
them. 

Discussion 

Although  differences  were  subtle  between  the  manner  in  which  subjects  were  directed  to  plan  for 
technology-supported  instruction  using  a traditional  versus  context-driven  approach,  there  appeared  to  be  definite 
differences  in  how  certain  aspects  of  their  plans  were  articulated.  This,  in  itself,  is  relatively  important  because  it 
illustrates  that  instructional  planning  by  practicing  teachers  can  be  affected  by  the  presence  of  specific  resources 
(Figure  4)  and  differences  in  planning  question  strategies. 

The  data  in  Table  1 indicate  that  plans  following  the  context-driven  model  included  many  more  elements  of 
an  effective  introduction.  A closer  examination  of  these  results  revealed  that,  in  describing  the  introductions  to  the 
proposed  instructional  programs,  nearly  twice  as  many  context-driven  plans  addressed  prerequisite  skills  and  made 
provisions  for  connecting  what  the  learners  were  about  to  learn  to  what  they  already  knew  how  to  do.  In  addition, 
more  than  twice  as  many  context-driven  plans  articulated  the  relevancy  of  the  new  skill  to  be  learned.  Inclusion  of 
these  important  Introduction  elements  would  most  likely  improve  the  overall  effectiveness  of  the  instructional 
program,  and  the  fact  that  the  context-driven  plans  facilitated  more  of  these  elements  indicates  that  “real”  contexts 
necessitate  more  effective  and  complete  introductory  experiences.  It's  likely  that  the  subjects  felt  the  T^-graders 
receiving  the  instruction  would  be  less  accustomed  to  a real-world  context  in  the  classroom  and  would  therefore  need 
more  information  up  front.  Conversely,  the  instructional  plans  following  the  traditional  approach  included  twice  as 
many  instances  of  the  Activity /Information  elements  dealing  with  relating  the  skills  to  be  learned  to  a clearly-defined 
content  domain,  providing  learner  guidance  for  the  application  of  the  information  presented  to  the  target  skills,  and 
breaking  down  the  process  of  performing  rules  into  steps.  It's  possible  that  provisions  for  the  more  directed  nature 
of  these  elements  were  not  considered  by  the  subjects  as  they  incorporated  those  less-directed  contexts  constituting  a 
majority  of  the  context-driven  plans:  creation,  simulation,  or  situation  exploration.  This  represents  an  important 
point  that  has  no  doubt  been  identified  by  developers  of  creation  or  simulation  environments:  How  much  guidance  is 
appropriate  within  these  types  of  environments? 

On  the  other  hand,  a closer  examination  of  the  elements  included  in  the  activity/information  component 
showed  that  the  context-driven  plans  described  many  more  opportunities  for  the  learners  to  explore  the  learning 
environment  with  minimal  instructor  guidance,  and  these  plans  also  offered  many  more  instances  of  examples  and 
nonexamples.  Again,  these  effective  Activity/Information  elements  were  most  likely  inherent  in  the  chosen  contexts, 
which  lends  support  to  the  value  of  structuring  these  types  of  environments.  However,  another  example  of  a 
potentially  harmful  use  of  more  student-directed  learning  environments  is  apparent  in  the  data  for  the 
Practice/Feedback  component.  Surprisingly,  most  of  the  plans  included  provisions  for  appropriate  practice  provided 
to  all  the  learners.  But  nearly  twice  as  many  instances  of  providing  initial  learner  guidance  within  the  practice  was 
identified  within  the  traditional  plans.  Initial  learner  guidance  is  an  important  element  of  effective  instruction 
(Hunter,  1982;  Merrill,  1994),  and  it  is  quite  possible  that  the  meaningful  learning  contexts  depicted  within  the 
context-driven  plans  were  conceptualized  by  the  subjects  as  requiring  or  demanding  less  direction  from  the 
instruction/instructor.  This  attitude  may  diminish  the  potential  effectiveness  of  successfully  implementing  such 
learning  environments  within  the  classroom,  and  may  in  fact  represent  some  of  the  reasons  why  teachers  often  have 
mixed  results  when  structuring  more  student-directed  contexts. 

Subjects  reported  many  fewer  elements  of  effective  reviews  or  transfer  opportunities,  and  this  was 
consistent  across  both  design  types.  Either  they  were  simply  tired  of  writing,  or  they  neglected  these  important 
elements  of  effective  instructional  design. 

The  biggest  difference  between  the  plans  following  the  traditional  versus  context-driven  approaches  was  the 
type  of  contexts  selected  and  described.  Thirty-three  context  descriptions  for  Creation,  Simulation,  Situation 
Exploration,  Communication,  and  Real  types  were  identified  in  the  traditional  plans,  while  51  of  these  contexts  were 
described  in  the  context-driven  plans.  These  contexts  generally  represent  more  student-directed  environments 
favored  by  constructivist  designers.  This  big  difference  in  frequencies  indicate  that  simply  encouraging  the  designer 
to  consider  how  the  skills  are  used  in  the  real  world  and  then  make  a context  selection  from  an  existing  list  can  have 
a dramatic  impact  on  the  type  of  context  selected.;  It  is  interesting  to  note  that,  without  such  prompting,  a good 
percentage  of  the  traditional  plans  communicated  more  tradition  uses  of  electronic  media  to  define  learning  contexts: 
Reference,  Game,  Drill  and  Practice,  Tutorial.  And  in  the  analysis  of  the  different  plans,  the  category  “Linear 
Information  Presentation”  needed  to  be  included  to  accommodate  those  traditional  plans  that  could  not  be 


categorized  into  one  of  the  other  context  types.  Perhaps  the  most  important  thing  about  this  data  is  that  no  use  of 
electronic  media  to  deliver  linear  instruction  was  identified  within  any  context-driven  plan. 

Perhaps  one  of  the  most  important  findings  within  the  data  analysis  was  the  role  computers  played  in 
encouraging  different  social  environments  in  the  classroom.  Eighteen  context-driven  plans  described  the  use  of 
computers  in  small  groups,  compared  with  only  six  context-driven  plans  for  individual  computer-based  instruction. 
This  was  nearly  the  opposite  of  the  traditional  plans  which  described  seven  instances  of  small  group  computer  use 
compared  with  14  plans  for  individual  computer-based  instruction.  While  individual  computer  use  can  prove  to  be  a 
very  effective  learning  situation,  the  benefits  of  grouping  students  around  computers  and  sharing  the  resources  may 
contribute  to  a more  effective  overall  learning  environment,  especially  in  classroom  where  computer  resources  are 
limited  (see  Johnson,  Johnson  and  Holubec,  1990). 

Taken  as  a whole,  the  results  from  this  study  offer  both  promise  and  insight  into  the  possibilities  of  utilizing 
context-driven  instructional  design.  The  limited  results  reported  in  this  study  indicate  that,  in  order  to  maximize  the 
potential  efficacy  of  a context-driven  design,  certain  learning  conditions  or  instructional  component  elements  must 
be  explicitly  addressed.  Providing  initial  learner  guidance  for  practice,  connecting  instructional  information  to  the 
target  skills,  relating  target  skills  to  a particular  content  domain,  and  breaking  down  the  process  of  learning 
intellectual  skills  into  steps  represent  important  aspects  of  the  learning  environment  that  may  not  be  inherent  within 
the  implementation  of  those  meaningful  learning  contexts  described. 

Most  of  the  subjects  reported  that  they  would  prefer  to  design  instruction  following  the  context-driven 
model.  They  also  felt  as  if  the  context-driven  plan  would  be  more  successful  in  the  classroom.  This  indicates  that  a 
context-driven  type  of  design  may  help  motivate  designers  to  develop  more  effective  and  creative  instructional 
programs. 

The  contrived  list  of  contexts  seemed  to  have  a profound  impact  on  context  selection.  Although  most 
instructional  design  models  shy  away  from  providing  such  contrived  lists  and  descriptions  of  media  use  or 
content/context  methodologies  (presumably  to  increase  the  generalizability  of  the  model),  it  may  benefit  novice 
designers  to  have  such  tools  at  their  disposal.  This  may  also  benefit  the  novice  designer  in  the  area  of  media 
selection.  It  is  interesting  to  note  that  computers  represented  the  medium  of  choice  within  most  of  the  designs. 

Some  contexts  (like  Communication)  are  more  dependent  on  computers,  but  it’s  possible  that  computers  were 
favored  because  teachers  and  novice  designers  lacked  the  training  and/or  experience  to  conceptualize  how  other 
media  might  be  used  to  facilitate  meaningful,  purposeful  learning  environments,  with  much  less  expense  involved. 

Results  from  this  simple  study  raise  a number  of  questions  suitable  for  future  research  consideration.  For 
example,  will  a context-driven  model  of  instructional  design  actually  result  in  a more  effective  instructional  program 
than  a more  traditional  ISD  model?  How  can  context-driven  versus  more  traditional  ISD  models  be  compared?  Is  it 
possible  to  determine  if  different  context  types  favor  the  natural  inclusion  of  specific  instructional 
components/elements?  Do  different  types  of  electronic  media  favor  different  context  types  or  context  functions? 
What,  if  any,  is  the  relationship  between  describing  how  a particular  skills  is  used  in  the  real  world  and  selecting  the 
type  of  context  to  be  cultivated  within  the  classroom  ? It  is  the  hope  of  the  author  that  the  quest  toward  answering 
these  types  of  questions  will  help  instructional  technologists  develop  more  productive,  innovative,  creative,  and 
effective  models  of  instructional  design. 


BEST  COPY  AVAILABLE 


ERIC 


303 


2SS 


Figure  1:  Basic  Instructional  Design  Model 


Component 

Elements 

Context 

Function7 

Introduction 

• Gain  learner  attention 

• Articulate  in  some  way  the  SKA1  already  needed  to  succeed  within  the  new  learning  environment 

• Identify  opportunities  in  which  learners  will  relate  what  is  about  to  be  learned  (goal)  to  what  they 

already  know  how  to  do 

• Inform  learners  of  objectives  (o.k.  to  be  vague) 

• Fit  objective(s)  into  a "Big  Picture"2 

• Present  the  utility  (relevance)  of  the  SKA  to  be  learned 

• Clearly  identify  the  incentives/re  wards  for  learning  the  SKA  and  succeeding  within  the  learning 

environment 

• Establish  clearly-perceived  learner  accountability,  role(s)  and  task(s)  within  the  learning 
environment 

• Establish  clearly-perceived  instructor  role(s)  and  learner  support  mechanisms 

• Employ  specific  content-area  methodologies  if  applicable  (ensuring  that  all  other  elements  are 

properly  addressed)3 

tp 

IS 

|ii| 

. . , 

Instruction 

Activities 

• Establish  in  context  the  appropriate  conditions  for  the  tvpets^  of  new  SKA  facilitated4 

• Relate  all  SKA  to  a clearly-defined  content  domain5 

• Provide  learner  guidance  for  learners  as  they  apply  information  presented  in  context  to 
SKA  being  facilitated 

• Present  a variety  of  clear,  concrete  examples  and  nonexamples  in  context 

• Provide  opportunities  for  learners  to  explore  the  learning  environment  with  minimal  instructor 

guidance  and  intervention 

Practice 

& 

Feedback6 

• Provide  initial  learner  guidance 

• Must  match  performances  and  conditions  indicated  within  objectives  and  presented  in  context 

• All  should  get  practice 

• Feedback  as  immediate  as  possible  (unless  delayed  feedback  is  desirable) 

Review 

• Provide  opportunities  for  learners  to  summarize  the  key  ideas  (including  how  these  ideas  fit  into  the 

"Big  Picture”),  what  they  learned  how  to  do,  and  how  they  personally  learned  it 

• Restate  objectives 

Assessment 

• Must  match  performances  and  conditions  indicated  within  objectives  (goal  may  be  only  thing 
assessed) 

Transfer 

• Present  new  context  which  elicits  the  same  performances  under  different  conditions 

• Make  the  utility  of  succeeding  within  the  new  context  apparent 

• Present  cues  within  the  new  context  which  aid  learners  in  selecting  and  applying  the  appropriate 

previously-learned  SKA 

• Clearly  identify  the  incentives/re  wards  succeeding  within  the  new  context 

J:.. 

Notes: 


1 SKA  refers  to  Skills,  Knowledge,  and  Attitudes.  Unless  otherwise  noted,  “SKA”  represent  content-domain  as  well  as  context-specific 
(including  social  interaction)  performances. 

2 A “Big  Picture”  often  consists  of  a text  and/or  graphic  representation  of  how  a goal’s  SKA  fit  into  a particular  context  (like  saving  the  rain 
forest,  meeting  a client’s  need,  running  a small  business),  a content  domain  (like  biology,  American  Literature,  project  management),  a 
cognitive-behavioral  domain  (like  problem-solving,  study  skills,  self-esteem,  physical  fitness),  and/or  social  domain  (like  cooperative  learning, 
team  building,  role-playing).  Big  Pictures  often  present  a bridge  between  what  the  learners  already  know  how  to  do  and  what  they  are  going  to 
learn  how  to  do. 

3 Examples  of  content-area  methodologies  include  “Math  Their  Way,”  Distar  Reading  Program,  and  biology  learning  cycles. 

4 The  manner  in  which  learning  activities  (including  information  presentation)  are  structured  depend  on  the  type  of  behavior  stated  in  each 
objective  (see  the  Classifying  Performances  and  Learning  Conditions  for  Different  Outcome  Types  charts). 

5 ’’Content  domain”  refers  to  the  general  body  of  knowledge  (information)  in  which  the  SKA  are  associated.  For  example,  music,  biology, 
football,  world  history,  and  team  building  all  represent  various  content  domains. 

6 Once  an  objective  has  been  established,  practice  and  feedback  are  the  most  important  elements  of  any  instruction.  The  most  important  things 
humans  learn  are  acquired  through  informal  information  presentation  and  lots  of  practice  and  feedback. 

7 These  context  functions  are  described  by  Tessmer  and  Richey  (1997)  as  context  “types.”  The  orienting  type  represents  a context  which 
precedes  the  learning  event  (information  and  examples  directed  at  the  target  SKA)  and  contains  factors  which  influence  the  learner’s  readiness  to 
learn  the  target  SKA.  The  instructional  type  represents  a context  in  which  conditions  specific  to  the  acquisition  of  the  target  SKA  are  presented. 
The  transfer  type  of  context  is  presented  after  learners  have  demonstrated  they  have  acquired  the  target  SKA.  Transfer  contexts  provide  an 
opportunity  for  learners  to  apply  SKA  to  conditions  outside  initial  learning  context.  It  is  important  to  identify  which  component(s)  are  addressed 
by  a chosen  context  type  because  it  helps  the  designer  determine  which  elements  of  the  learning  environment  need  to  be  presented  before, 
during,  or  after  the  learners  experience  the  events  associated  with  the  context. 

The  ID  components  and  elements  presented  in  this  chart  are  based  on  Gagn6  & Driscoll  (1988),  Keller,  (1987),  Merrill  & Tennison 
(1994),  Sullivan  & Higgins  (1983),  and  Tessmer  & Richey  (1997). 
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Figure  2.  Classifying  Performances 


Category 

Description  of  Performance  Outcome 

Common  Verbs 

Verbal 

Information 

(Declarative  Knowledge) 

Verbal  information  the  students  must  state,  including:  facts,  dates,  people,  names, 
principles,  generalizations  etc. 

State,  Recite, 

Tell,  Declare,  Name,  List,  Define 
[Bloom:  Knowledge ]‘ 

Intellectual  Skills : 
(Procedural  Knowledge) 
Discriminations 

Distinguishing  objects,  features,  or  symbols  (smooth  versus  not  smooth,  for 
example) 

Distinguish, 

Differentiate, 

Intellectual  Skills : 
Concepts 

Concrete  Concepts: 

Objects  (pans  of  the  body,  for  example).  Classes  of  objects  (plants,  cells,  etc.). 
Object  features  (5  arms,  red,  etc.),  and  Object  relations  (above,  near,  etc.) 

... that  can  be  pointed  out  and  identified. 

Defined  Concerns: 

Objects,  principles,  classes,  features,  and  relations  that  cannot  be  identified  by 
pointing  them  out.  They  must  be  defined. 

Identify,  Label 
Classify  instances. 

Sort, 

Categorize 

[Bloom:  Comprehension ] 

Intellectual  Skills: 
Rules 

Rules  make  it  possible  to  do  something  using  symbols  (most  commonly,  the 
symbols  of  language  and  math). 

Rules  include  the  application  of  single  principles  to  explain,  describe,  or  predict 
phenomena  or  events. 

Rules  make  it  possible  for  students  to  respond  to  a class  of  things  with  a class  of 
performances. 

Solve,  Show,  Demonstrate, 
Generate,  Develop,  Create, 
Determine.  Calculate,  Predict 
[Bloom:  Application ] 

Intellectual  Skills: 
Higher-Order  Rules 
( Problem  Solving) 

Higher  order  rules  employ  more  than  one  rule  or  principle  to  solve  problems, 
perform  tasks,  or  explain,  describe,  and  predict  phenomena  or  events. 

Students  must  decide  which  rules  or  principles  must  be  utilized  to  perform  tasks 
or  explain,  describe,  or  predict  phenomena  or  events. 

Solve,  Show,  Demonstrate,  General 
Develop,  Create,  Determine, 
Calculate,  Predict,  Defend,  Support 
[Bloom:  Analysis,  Synthesis, 
Evaluation ] 

Motor  Skills 

Motor  skills  represent  physical  activities  requiring  movement  and  coordination  of 
all  or  part  of  the  body. 

Execute,  Perform,  Swim,  Walk,  Run 
Climb,  Drill,  Saw 

Attitudes 

Attitudes  represent  intrinsically  motivated  choices  people  make.  Some  of  the 
most  important  outcomes  are  really  attitudes. 

Choose,  Decide,  Participate 
[Bloom:  Evaluation  ] 

‘Schools  often  use  Bloom’s  Taxonomy  of  Educational  Objectives  (1984)  as  a basis  for  classifying  outcomes  and  standards  in  an  effort  to  ensure 
that  skills  at  variety  of  cognitive  levels  are  being  addressed. 
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Figure  3.  Learning  Conditions  for  Different  Outcome  Types 


Performance  Type 

Strategies  or  Conditions  for  Presenting  Instructional  Events 

Verbal  Information 

• Use  the  term  or  definition  in  a sentence. 

• Relate  the  information  (term  or  definition)  to  preexisting  knowledge. 

• Present  all  terms  clearly  using  the  fewest  number  of  words  to  convey  the  meaning.  If  more  than  five  terms 
or  units  of  information  are  to  be  presented  in  one  lesson,  group  related  terms  or  units  into  five  or  fewer  clearly  defined 
categories. 

• Use  a variety  of  concrete  (observable)  examples  when  possible,  emphasizing  the  clear  and  well-defined 
features  that  relate  directly  to  the  information. 

• Explain  clearly  how  learners  will  be  expected  to  recall  the  information.. 

• Make  information  readily  accessible  to  learners,  and  provide  opportunities  for  them  to  explore  “nice-to- 
know”  information  associated  with  the  knowledge 

• Practice  with  immediate  feedback! 

Intellectual 

Skills 

• Present  varied  examples  or  instances  of  concepts  and  rule  applications,  calling  attention  to  the  distinctive 
features  of  examples,  definitions,  and  procedures. 

• Present  nonexamples  or  noninstances  of  the  concept  if  they  will  help  to  clarify  the  concept. 

• Encourage  learners  to  recall  previously  learned  information  or  examples  that  illustrate  concepts  or  rules 
being  presented. 

• Clearly  communicate  the  definition  of  defined  concepts,  using  the  fewest  words. 

• Break  down  the  process  of  performing  or  applying  rules  into  steps,  and  clearly  communicate  these  steps  to 
the  students. 

• Demonstrate  an  application  of  the  rule  for  the  students. 

• Provide  learners  with  opportunities  to  ,lplay”  with  concepts  and  rules  within  simulated  or  “real” 
environments,  identifying  and  selecting  their  own  examples  and  nonexamples  of  concepts  and  rule  applications  if 
possible. 

• Present  a variety  of  contexts  or  experiences  that  allow  the  students  to  practice  applying  the  rules  or 
identifying/describing  concepts,  providing  guidance  throughout  early  stages  of  practice. 

Attitudes 

• Clearly  identify  examples  of  choices  made  by  people  who  possess  the  desired  attitude  (credible  and 
attractive-similarity,  familiarity,  appearance). 

• Clearly  identify  instances  in  the  students’  lives  in  which  making  choices  are  based  on  the  attitude  being 
presented. 

• Make  students  aware  of  the  personal  benefits  gained  by  making  choices  based  on  attitudes  (preferably  by 
someone  the  students  admire). 

• Allow  students  the  opportunity  to  practice  making  choices  associated  with  the  desired  attitude  (role- 
playing,  group  discussion,  etc.)  and  give  them  feedback. 

Motor  Skills 

• Verbally  guide  learners  through  routine. 

• Visually  present  example  of  routine  execution. 

• Practice  with  immediate  feedback. 

• Encourage  the  use  of  mental  practice. 

Figure  4.  Classifying  Electronic  Media-Supported  ", Educational " Contexts 


Context 

Description 

Creation: 

This  type  of  context  provides  opportunities  for  learners  to  create  something.  [O.I.T]1 

Simulation: 

This  context  type  allows  the  learner  to  make  decisions  in  the  development  and  subsequent  operation  of  a 
simulated  environment  or  situation.  Simulations  often  try  to  replicate  real-world  environments.  This  type  of 
context  is  often  "problem-solving"  in  nature.  [O,  I,  T] 

Situation  Exploration: 

Unlike  a simulation  context,  this  type  of  context  does  not  allow  the  learner  to  make  decisions  regarding  the 
development  of  a simulated  environment,  but  the  learner  can  freely  explore  within  a simulated  environment  or 
situation.  This  type  of  context  is  often  "problem-solving"  as  well  case-based.  [I,  T] 

Reference  Exploration: 

This  context  type  allows  the  learner  to  freely  explore  and  access  reference-type  information.  [I,  T] 

Tutorial 

(Direct 

instruction): 

This  context  type  generally  presents  "new"  information  (usually  in  a linear  or  stepwise  format),  and  either 
provides  a certain  degree  of  practice  using  the  information  in  some  way,  or  applies  the  SKA  to  specific 
example(s).  [I] 

Linear  Information 
Presentation 
(No-practice  tutorial): 

This  context  type  simply  provides  the  learners  with  the  linear  presentation  of  information  and  examples.  That’s 

it.  m 

Drill  and  Practice: 

Generally,  this  type  of  environment  does  not  present  "new”  information,  but  provides  practice  and  feedback  over 
specific  skills  (often  knowledge,  defined  concepts  and  rules).  [I] 

Game: 

This  type  of  context  usually  engages  learners  in  competition,  cooperation,  puzzles,  or  strategies,  often  for  the 
sake  of  entertainment.  Other  contexts  may  employ  this  context  to  because  of  the  motivational  advantages  of 
games.  [0,  T] 

Communication: 

This  context  allows  learners  to  communicate  with  other  people  via  text,  audio,  binary  files,  and/or  video 
information.  [O,  I,  T] 

Real: 

Contexts  in  this  category  could  fall  into  any  of  the  above  groups,  or  none.  The  distinguishing  characteristics  for 
these  contexts  are  simply  that  they  constitute  real-world  situations  and  settings  (home,  school,  work,  play).  [O,  I, 
T] 

lIn  general,  each  context  type  lends  itself  to  one  or  more  ID  context  functions.  O = orienting  function,  I = instructional  function,  T = transfer 
function. 
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Table  1.  Total  Instructional  Component  Element  Occurrences  by  ID  Type  and  Standard 


Number  of  Component  Elements  Described 


Instructional  Component 

Standard  7. 1 0 
(Maps) 

Standard  7.8  (Compare) 

Total 

Trad. 

ID 

Context 

ID 

Trad. 

ID 

Context 

ID 

Trad. 

ID 

Context 

ID 

Introduction 

27 

46 

38 

52 

65 

98 

Activity/Information 

45 

46 

48 

57 

93 

103 

Practice/Feedback 

30 

27 

28 

35 

58 

62 

Review 

20 

17 

15 

14 

35 

31 

Assessment 

9 

B 

13 

7 

22 

15 

Transfer 

10 

8 

5 

7 

15 

15 

Total 

141 

152 

147 

172 

288 

324 

292 


Table  2.  Context  Type  Selected  by  Electronic  Medium  and  Scenario 


Standard  7.10  (Maps) Standard  7.8  (Compare) Total 


Context  Type 

Media1 

Trad. 

ID 

Context 

ID 

Trad. 

ID 

Context 

ID 

Trad. 

ID 

Context 

ID 

c 

3 

5 

2 

4 

5 

9 

1 : Creation 

V 

0 

0 

0 

0 

0 

0 

OHP 

0 

0 

0 

0 

0 

0 

A 

0 

0 

0 

0 

0 

0 

C 

1 

1 

1 

2 

2 

3 

2:  Simulation 

V 

0 

1 

1 

0 

1 

1 

OHP 

0 

0 

0 

0 

0 

0 

A 

0 

0 

0 

0 

0 

0 

C 

4 

7 

2 

6 

6 

13 

3:  Situation  Exploration 

V 

3 

1 

1 

2 

4 

3 

OHP 

0 

0 

0 

0 

0 

0 

A 

0 

0 

0 

0 

0 

0 

C 

5 

2 

6 

5 

11 

7 

4:  Reference 

V 

3 

0 

0 

2 

3 

2 

Exploration 

OHP 

0 

0 

0 

0 

0 

0 

A 

0 

0 

0 

0 

0 

0 

C 

4 

3 

3 

2 

7 

5 

5:  Tutorial  (Direct 

V 

0 

0 

0 

0 

0 

0 

Instruction) 

OHP 

0 

0 

0 

0 

0 

0 

A 

0 

0 

0 

0 

0 

0 

C 

6 

3 

4 

2 

10 

5 

6:  Drill  and  Practice 

V 

1 

0 

0 

0 

1 

0 

OHP 

0 

0 

2 

0 

2 

0 

A 

0 

0 

1 

0 

1 

0 

C 

4 

2 

1 

1 

5 

3 

7:  Game 

V 

0 

0 

1 

0 

1 

0 

OHP 

0 

0 

0 

0 

0 

0 

A 

0 

0 

0 

0 

0 

0 

C 

0 

2 

0 

5 

0 

7 

8:  Communication 

V 

0 

0 

0 

0 

0 

0 

OHP 

0 

0 

0 

0 

0 

0 

A 

0 

0 

0 

0 

0 

0 

C 

1 

3 

0 

2 

1 

5 

9:  Real 

V 

0 

0 

0 

0 

0 

1 

OHP 

0 

0 

0 

0 

0 

0 

A 

0 

0 

0 

0 

0 

0 

C 

4 

0 

1 

0 

5 

0 

10:Linear  Information 

V 

1 

0 

1 

0 

2 

0 

Presentation 

OHP 

1 

0 

0 

0 

1 

0 

A 

1 

0 

1 

0 

2 

0 

*C  = Computer,  V - Video,  OHP  = Overhead  Projector,  A = Audio 
Table  3:  Social  Interaction  Environments  by  Electronic  Media 


Standard  7. 10  Standard  7.8  (Compare) 

(Maps) Total 


Electronic  Medium 

Interact 

Trad. 

ID 

Context 

ID 

Trad. 

ID 

Context 

ID 

Trad. 

ID 

Context 

ID 

Class 

I 

0 

0 

2 

1 

2 

Computers 

Group 

3 

10 

4 

8 

7 

18 

Indv 

13 

8 

15 

5 

14 

6 

Class 

2 

1 

5 

2 

7 

3 

Video 

Group 

1 

2 

2 

3 

3 

5 

Indv 

1 

0 

1 

2 

2 

2 

Class 

1 

1 

0 

0 

1 

1 

Overhead  Projector 

Group 

0 

0 

1 

0 

1 

0 

Indv 

0 

0 

0 

0 

0 

0 

Class 

0 

0 

1 

0 

1 

0 

Audio 

Group 

0 

0 

0 

0 

0 

0 

Indv 

0 

0 

0 

0 

0 

0 
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Environments 
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Abstract 

This  study  investigated  the  effects  of  “ instructionist”  ( navigational , functional)  versus  “constructionist” 
(adaptive)  World  Wide  Web  (WWW)  site  environments  on  collaboration , achievement , attitudes , perceived 
level  of  learner  controls  reported  by  students  after  working  in  cooperative  dyads  within  the  different  types 
of  web-based  learning  environments . 

Three  different  versions  of  an  instructional  WWW  site  were  developed , corresponding  to  the  different  types 
of  interaction  strategies.  Twelve  8th -grade  students  (6  dyads)  were  each  assigned  three  similar 
instructional  problems  to  solve , and  they  worked  within  the  different  web  environments  to  accomplish  their 
instructional  tasks.  When  the  dyads  experienced  the  navigational  site , they  viewed  a main  menu  with 
broad  topics  displayed  as  hypertext  links.  Selecting  a particular  link  displayed  a listing  of  specific 
informational  pages  related  to  the  topic.  This  type  of  environment  was  classified  as  “instructionist”  due  to 
its  relatively  low  degree  of  learner  interactivity  and  the  inherently  linear  format  of  the  information 
presentation.  When  the  dyads  were  assigned  to  the  functional  site,  they  began  their  exploration  with  a site - 
specific  search  engine.  After  entering  one  or  more  terms,  the  search  engine  displayed  topic  pages  and/or 
specific  information  pages  related  to  the  terms  submitted.  When  the  dyads  experienced  the  adaptive 
(constructionist)  site,  they  were  instructed  to  construct  a web  site  that  addressed  the  instructional  problem. 
Preliminary  results  indicated  a number  of  differences  between  the  subjects'  performance  and  behavior 
while  experiencing  the  adaptive  site  (construction)  versus  the  other  two  treatments.  These  included 
spending  more  time  with  the  material,  reporting  a higher  level  of  interest  in  the  instructional  material, 
trying  harder  to  learn  the  material,  reporting  a higher  degree  of  learner  control,  a higher  perception  of 
interactivity,  and  an  increase  in  the  amount  of  positive  interpersonal  interaction. 

Recommendations  for  future  research  are  also  included. 

Introduction 

“Interactivity”  as  it  relates  to  computer-based  instruction  (CBI)  is  commonly  defined  as  the  degree  to  which 
user  input  affects  message  presentation.  With  the  emergence  of  the  WWW  as  a viable  and  somewhat  homogeneous 
computer-mediated  environment,  distinct  types  of  web-page  interactivity  have  emerged.  Web  pages  may  include 
interactive  strategies  classified  as  navigational,  functional,  or  adaptive  (Guay,  1995).  The  characteristics  of  these 
different  website  design  strategies  follow: 


Type  of  Web  Interactivity  j 

i 

1 

Description 

Relative  Degree 
of  Interactivity 

Navigational 

(instructionist) 

| • Focus  on  navigating  information  space 
| • Hypertext  menus  and/or  individual  links 
• Control  limited  to  what  user  can  access  “next” 

Low 

Functional 

• User  interacts  with  system  to  accomplish  goal  by  making  decisions  about 

what  information  is  available  on  a page 

• User  receives  feedback  throughout  interaction 

• Common  examples  include  search  engines,  games,  accessing  databases 

Medium 

Adaptive 

(constructionist) 

• Similar  to  functional  site,  except  user  adapts  information  space  to  meet 
specific  needs  or  desires 
I • User  “authors”  information  site 

High 

The  “navigational”  web  environment  is  loosely  classified  as  instructionist  due  to  its  relatively  low  degree  of 
learner  interactivity  and  the  more  structured  nature  of  the  information  organization  and  presentation.  Papert  (1993) 
describes  instructionist  learning  environments  as  those  learning  situations  subscribing  to  a teacher-centered  paradigm 
in  which  students  assume  the  role  of  information  recipient.  He  contrasts  “instructionism”  with  what  he  refers  to  as 
“constructionism,”  which  represents  learner-centered  educational  experiences  focusing  on  student  creations  or 
constructions.  As  students  construct  things  in  the  real  world  (or  the  classroom),  these  experiences  support  the 
construction  of  knowledge  structures  in  the  head.  Constructionism,  in  a sense,  represents  operationalized 
constructivism. 

Adaptive  web  environment  can  be  classified  as  “constructionist”  due  to  their  relatively  high  degree  of 
learner  interactivity  and  the  fact  that  learners  have  an  opportunity  to  create  their  own  web  environments.  Adaptive 
web  experiences  range  from  using  web  editors  to  create  web  pages  and  sites,  to  more  structured  creation  strategies  in 
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which  users  make  specific  design  decisions  about  existing  web  pages  (for  example,  see  the  adaptive  features  used 
within  the  websites  described  by  de  La  Passardiere  & Dufresne,  1992;  and  Eklund,  1996).  As  it  becomes  easier  to 
create  web  pages  and  sites,  opportunities  for  the  use  of  adaptive  web  experiences  within  instructional  settings  are 
increasing. 

This  exploratory  study  was  designed  to  examine  the  effects  of  navigational,  functional,  and  adaptive  web- 
based  learning  experiences  on  a number  of  dependent  variables,  including  achievement,  attitudes,  and  perceived 
level  of  learner  control. 

Method 


Part  of  the  exploratory  nature  of  this  study  involved  the  development  of  different  versions  of  the 
instructional  websites  corresponding  to  the  different  types  of  interaction  strategies.  The  first  step  in  the  instructional 
web  development  process  involved  the  creation  of  a website  that  contained  individual  pages  with  content  that  could 
be  categorized  in  a number  of  ways.  “Battles  of  the  Civil  War”  was  chosen  as  the  topic  for  this  study  because  the 
subject  matter  lent  itself  to  the  subject  pool  (junior  high  school  and  high  school  students),  and  because  the  topic  was 
broad  enough  to  support  the  different  types  of  web  environments  under  development.  The  following  chart 
summarizes  the  main  characteristics  of  each  web  type  developed: 


Web  Type 

Characteristics 

Navigational 

This  type  of  web  environment  consisted  of  an  introduction  page  that  displayed  the  information  about  one  of  the 
selected  battles  (for  example,  the  battle  of  Shiloh  was  the  first  battle  studied  by  all  the  groups).  A link  on  this  page 
led  to  a main  menu  page  with  broad  topics  displayed  as  hypertext  links.  Selecting  a particular  link  displayed  a 
listing  of  specific  informational  pages  related  to  the  topic.  See  Figure  One  for  a sample. 

Functional 

This  type  of  web  environment  consisted  of  an  introduction  page  that  displayed  the  information  about  one  of  the 
selected  battles.  A link  on  this  page  led  to  a search  page  that  required  the  inputting  of  terms  to  display  links  to 
specific  informational  pages  related  to  the  terms  selected.  Subjects  could  also  choose  how  the  links  to  the 
requested  term(s)  would  be  displayed:  alphabetically,  by  timeline,  by  location,  by  people  involved,  and  by  battle 
event.  See  Figure  Two  for  a sample. 

Adaptive 

In  this  environment,  the  students  were  directed  to  the  same  introduction  and  search  pages  as  displayed  in  the 
Functional  group.  They  were  then  directed  to  construct  their  own  website  that  could  help  other  students  (and 
themselves)  learn  the  information  needed  to  successfully  answer  questions  about  the  battle.  All  students  had 
experience  using  Netscape  Composer  to  create  local  webpages.  The  students  were  informed  that  they  could  use 
any  resources  on  the  web,  but  that  the  pages  found  within  the  local  Civil  War  site  did  include  an  adequate  amount 
of  basic  information. 

The  subjects  for  this  study  were  twelve  S^-grade  students  who  were  participating  in  an  afterschool 
enrichment  program  at  a middle  school  in  a small  rural  Midwestern  town.  These  students  volunteered  to  participate 
in  the  enrichment  program,  which  usually  involved  a good  deal  of  extra  time  spent  on  the  school’s  computers  using 
the  Internet  for  independent  research.  Most  of  the  time,  these  students  worked  in  pairs  or  triads  to  complete  assigned 
group  project  work.  Seven  females  and  nine  males  participated  in  this  study. 

During  the  fust  afterschool  meeting  constituting  the  study,  the  subjects  were  informed  that  they  were  going 
to  study  a web-based  unit  of  instruction  about  some  of  the  important  Civil  War  battles.  They  were  told  that 
successfully  completing  these  units  of  study  would  help  them  perform  better  on  the  standardized  history  questions  to 
be  presented  on  their  standardized  tests  administered  at  the  end  of  the  year.  They  were  also  informed  that  their 
performance  on  the  unit  quizzes  would  be  ranked,  and  the  highest-ranking  student  at  the  end  of  the  unit  would 
receive  a special  Civil  War  prize.  The  subjects  were  also  informed  that  they  were  going  to  be  randomly  assigned  to 
cooperative  dyads  for  the  unit. 

The  subjects  were  informed  that  the  Civil  War  unit  would  cover  three  important  battles:  the  battle  of  Shiloh, 
the  battle  of  2nd  Manassas  (the  second  battle  of  Bull  Run  ),  and  the  battle  of  Gettysburg.  The  subjects  were  also 
informed  that  the  information  to  be  studied  was  web-based,  and  that  they  would  study  each  battle  in  a different  way 
during  the  unit.  During  the  first  week,  all  groups  studied  the  battle  of  Shiloh.  Two  groups  were  randomly  assigned 
to  one  of  the  three  different  types  of  web  environments  (navigational,  functional,  or  adaptive). 

No  time  limits  were  imposed  on  any  of  the  groups.  The  students  met  regularly  for  one  and  a half  hours, 
three  times  per  week.  The  students  were  told  that  they  could  take  the  quiz  whenever  they  felt  ready.  When  a group 
indicated  that  they  were  finished  studying  their  assigned  battle,  they  were  administered  a brief  attitude  questionnaire 
followed  by  a 20-question  multiple-choice  quiz.  After  taking  the  survey  and  quiz,  they  were  assigned  to  one  of  the 
other  battles  under  a different  web-based  condition,  according  to  the  schedule  below: 


O 

ERIC 
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Navigation 

Functional 

Adaptive 

Shiloh 

1,4 

3,5 

2,6 

Manassas 

2,5 

1,6 

3,4 

Gettysburg 

3,6 

2,4 

1,5 

The  students  worked  in  a small  computer  lab  in  a comer  of  the  school’s  media  center.  Although  the  lab 
contained  15  computers,  only  six  were  turned  on,  with  the  two  members  of  each  dyad  sharing  a single  computer. 
After  all  the  students  had  studied  all  the  battles,  the  quiz  scores,  time,  and  attitude  survey  data  were  tabulated. 

Results 


The  mean  quiz  scores  for  each  battle  by  instructional  web  type  are  presented  in  Table  One.  These  data 
indicate  that  the  mean  score  for  all  12  subjects  on  the  battle  of  Shiloh  quiz  was  14.1,  the  mean  score  for  the 
Manassas  quiz  was  15.5,  and  the  mean  score  for  the  Gettysburg  quiz  was  16.6  out  of  a possible  20  correct.  Table 
One  also  presents  the  quiz  scores  by  instructional  web  type.  These  data  indicate  that  the  mean  quiz  score  for  the  12 
students  experiencing  the  navigational  web  type  was  14.4,  the  mean  score  for  all  subjects  experiencing  the  functional 
web  type  was  14.9,  and  the  mean  quiz  score  for  students  experiencing  the  adaptive  web  type  was  16.9. 

Table  Two  presents  the  average  amount  of  time  spent  experiencing  the  different  battle  webs.  These  data 
indicate  that  the  mean  time  for  all  12  subjects  studying  the  battle  of  Shiloh  web  was  89  minutes,  the  mean  time 
studying  the  Manassas  web  was  116  minutes,  and  the  mean  time  experiencing  the  Gettysburg  web  was  1 12  minutes. 
Table  Two  also  presents  time  by  instructional  web  type.  These  data  indicate  that  the  mean  time  spent  by  the  12 
students  experiencing  the  navigational  web  type  was  61  minutes,  the  mean  time  spent  by  all  subjects  experiencing  the 
functional  web  type  was  67  minutes,  and  the  mean  time  spent  by  students  experiencing  the  adaptive  web  type  was 
190  minutes. 

Table  Three  presents  the  mean  attitude  item  responses  by  type  of  web.  Many  of  the  responses  to  these 
items  were  very  similar  across  web  types,  but  some  of  the  items  appeared  to  have  a wide  variation  in  responses. 

After  experiencing  the  navigational  web,  subjects  appeared  to  strongly  agree  with  the  statement  “The  structure  of  this 
website  encouraged  me  to  explore  all  the  information  presented  about  my  assigned  Civil  War  battle”  (3.6).  In 
contrast,  after  experiencing  the  functional  website  the  subjects  reported  a 2.4  on  this  item,  indicating  that  they  did  not 
agree  with  the  statement  as  strongly.  Subjects  experiencing  the  functional  website  also  reported  relatively  lower 
agreement  with  the  statements  “I  had  control  over  what  I learned  related  to  the  Civil  War  while  using  this  Civil  War 
websites”  (2.8)  and  “I  am  confident  that  I will  do  well  on  a quiz  over  this  battle”  (2.8).  The  data  in  Table  Three  also 
indicate  that  the  subjects  reportedly  agreed  more  strongly  with  a number  of  statements  after  experiencing  the 
adaptive  website  relative  to  the  other  two  types  of  web-based  experiences.  These  statements  included 

“Learning  about  this  Civil  War  battle  was  interesting”  (3.7),  “I  had  control  over  what  I learned  related  to  the 
Civil  War  while  using  this  Civil  War  websites”  (3.8),  “I  had  control  over  the  Civil  War  information  presented  on  the 
computer”  (3.8),  “I  am  confident  that  I will  do  well  on  a quiz  over  this  battle”  (3.7),  and  “I  tried  hard  to  learn  the 
information  presented  about  this  battle”  (3.8). 

When  asked  to  rate  the  level  of  interactivity  each  type  of  web  experienced  offered,  subjects  reported  less 
interactivity  in  the  navigational  experience  (1.6),  relatively  moderate  amount  of  interactivity  in  the  functional  web 
experience  (2.2),  and  relatively  higher  interactivity  in  the  adaptive  web  experience  (2.8).  These  data  are  also 
reported  in  Table  Three. 

Discussion 

Due  to  the  limited  number  of  subjects  participating  in  this  pilot  study,  not  many  conclusions  can  be  drawn 
about  differences  in  performance  and  attitudes  between  subjects  assigned  to  the  different  treatment  groups. 

However,  based  on  the  preliminary  data  collected,  a number  of  observations  and  questions  seem  worthy  of  further 
discussion  and  subsequent  investigation.  For  example,  subjects  appeared  to  score  higher  on  those  quizzes  that 
addressed  information  over  the  battles  for  which  they  developed  websites  (adaptive).  The  following  observations 
and  attitudinal  responses  may  help  explain  why: 
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Time 

Subjects  spent  much  more  time  with  the  material  as  they  developed  their  websites. 

Motivation 

Subjects  reported  that  they  were  more  interested  in  learning  about  their  “adaptive”  battle. 

Effort 

Subjects  reported  that  they  tried  harder  to  learn  about  their  “adaptive”  battle. 

Control 

Subjects  reported  that  they  felt  as  if  they  had  more  control  over  the  information  presented  as  well  as  more  control  over 
their  own  learning  while  creating  their  battle  website. 

Interactivity 

Subjects  reported  that  they  felt  the  adaptive  experience  was  more  interactive  than  the  other  experiences.  Also,  the 
additional  time  spent  working  together  on  creating  the  website  increased  the  amount  of  time  the  dyad  members  spent 
interacting  with  each  other. 

All  of  the  factors  described  above  may  have  influenced  the  amount  of  information  “learned”  from  the  Civil 
War  battle  site.  Certainly,  a positive  relationship  should  exist  between  the  amount  of  time  spent  with  a topic  and  the 
recall  of  information  associated  with  it.  In  the  same  vein,  the  adaptive  website  did  require  a lot  more  effort  on  the 
part  of  the  learners  to  understand  and  organize  the  material  well  enough  to  generate  an  informational  web  page  about 
the  assigned  topic.  And  given  the  probable  personality  type  of  students  who  would  choose  to  participate  in  a 
voluntary  afterschool  enrichment  program,  an  increase  in  interest  for  a topic  requiring  a lot  more  “work”  isn’t 
surprising.  It  is  also  possible  that  the  construction  assignment  within  the  adaptive  experience  provided  a meaningful 
context  for  learning  about  the  relatively  abstract  content  associated  with  a war  fought  over  200  years  ago.  It  would 
be  interesting  to  see  if  similar  results  are  obtained  from  students  across  a wider  range  of  academic  ability  and 
experience. 

Perhaps  the  most  intriguing  aspect  of  these  VERY  LIMITED  and  PRELIMINARY  findings  is  the  notion 
that  differences  in  perceived  learner  control  as  well  as  interactivity  may  have  influenced  learning.  Although  degree 
of  interactivity  has  not  been  a research  variable  in  many  studies  involving  CBI,  a related  concept  that  has  been 
investigated  by  researchers  is  “learner  control.”  Learner  control  within  computer-mediated  environments  has  been 
an  important  instructional  design  variable  during  the  past  15  years  of  instructional  technology  research.  Learner 
control  has  been  defined  and  operationalized  differently  throughout  a number  of  studies  (see  Reeves,  1993;  Ross,  et 
al,  1989),  and  findings  from  various  learner  control  studies  are  mixed  (see  Friend  & Cole,  1990;  Large,  1996). 
Although  learner  control  strategies  have  varied  within  different  research  studies,  one  factor  that  has  helped  to  delimit 
all  types  of  learner  control  is  “interactivity.”  In  the  case  of  a number  of  learner  control  studies,  the  users  could 
choose  to  access  specific  types  of  information  related  to  elements  of  an  instructional  program.  These  elements 
included  additional  examples,  ancillary  information,  additional  practice  items  and  specific  types  of  feedback  (see 
Hannafin  & Sullivan,  1995;  Lee  & Lee,  1991;  Pridemore  & Klein,  1993).  A closer  examination  of  different  learner 
control  studies  reveals  that  none  explicitly  manipulated  variables  directly  related  to  degrees  of  interactivity,  and  very 
few  studies  included  data  indicating  the  perceived  level  of  control  learners  reported  after  interacting  with  the 
computer-mediated  learning  environment  (Schwier,  1993).  The  relationship  (if  any)  between  perceived  learner 
control,  degree  of  interactivity,  and  learning  would  be  an  excellent  focus  for  subsequent  investigation! 

Another  important  factor  related  to  interactivity  that  presumably  has  an  impact  on  learning  within 
cooperative  dyads  is  the  type  of  social  interactions  occurring  between  group  members.  Originally,  each  dyad  in  this 
study  was  going  to  be  videotaped,  and  the  nature  of  group  member  interaction  was  going  to  be  examined.  The 
school  district  where  this  study  took  place  had  a policy  against  videotaping  students;  consequently,  social 
interactions  were  not  studied.  In  future  studies,  group  member  interaction  will  be  closely  observed,  and  the 
following  types  of  interactions  will  be  documented: 

• Shared  the  mouse  and/or  keyboard 

• Helped  explain  information 

• Helped  summarize  the  information  presented 

• Helped  discern  relevant  information 

• Asked  for  help 

• Collaborated  on  making  navigation  decisions 

• Did  not  respond  to  solicitation  for  help 

• Checked  for  partner’s  understanding 

• Encouraged  partner 

• Off-task 

Perhaps  the  most  important  elements  provided  by  “constructionist”  learning  environments  like  the  adaptive 
type  employed  in  this  study  are  more  dynamic  and  constructive  social  interactions  within  a meaning  and  purposeful 
learning  context.  Productive  social  interactions  between  group  members  (as  well  as  human-computer  interaction) 
may  be  a key  element  for  deeper  cognitive  processing,  more  motivation  to  learn,  and  more  effort  employed  by  the 
learners.  If  this  is  true,  then  the  additional  time  needed  may  be  justified. 
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Figure  One.  “ Navigational  ” Website  Sample 


The  American  Civil  War 


Directions:  Next  week,  you  will  be  given  a quiz  over  the  Battle  of  Shiloh,  one  of  the  important  Civil  War  battles  fought 
in  Tennessee.  This  quiz  will  include  multiple-choice  questions  about  the  following  aspects  of  the  battle: 


1.  Dates 

3.  Impact  of  battle 

• Timeline  of  events  leading  up  to  battle 

4.  People 

• Timeline  of  battle  events 

• Timeline  of  events  following  the  battle 

• Military  leaders 

• Other  important  figures 

2.  Battle 

5.  History 

• Location(s) 

• Detailed  Battle  Timeline 

• Official  reports 

• Events 

• Historical  reflections 

• Outcome 

• Historical  commentary 

Click  the  arrow  below  to  access  a main  menu  of  links  to  access  specific  information  about  the  Battle  of  Shiloh. 

O 

AfttlbkUn... 


Battle  of  Shiloh  Main  Menu 

Dates 

Timeline  of  events  leading  up  to  battle 

Timeline  of  battle  events 

Timeline  of  events  following  the  battle 

Battle  Info 

Location(s) 

Detailed  Battle  Timeline 

Events 

Outcome 

Impact  of  battle 

People 

Military  leaders 
Other  important  figures 

History 

Official  reports 
Historical  reflections 
Historical  commentary 


m 

Back  to  Battle  of  Shiloh  Introduction  Page 
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Figure  2.  “ Functional  ” Website  Sample 


The  American  Civil  War 


Directions:  Next  week,  you  will  be  given  a quiz  over  the  Battle  of  Shiloh,  one  of  the  important  Civil  War  battles  fought  in 
Tennessee.  This  quiz  will  include  multiple -choice  questions  about  the  following  aspects  of  the  battle: 


1.  Dates 

3.  Impact  of  batde 

• Timeline  of  events  le  ading  up  to  batde 

4.  People 

• Timeline  of  batde  events 

• Timeline  of  events  following  die  batde 

• Military  leaders 

• Other  important  figures 

2.  Batde 

5.  History 

• Locadon(s) 

• Detailed  Batde  Timeline 

• Official  reports 

• Events 

• Historical  reflections 

• Outcome 

• Historical  commentary 

Click  the  arrow  below  to  access  a page  that  will  allow  you  to  search  and  display  pages  which  MAY  include  die  specific 
information  about  the  Battle  of  Shiloh  that  you  wish  to  access. 


e 


Battle  of  Shiloh  Search  Page 


Search  for:[" 


k Start  Search  v.  Reset 


Group  and  display  die  results  of  your  search  by  die  following  categories  (Check  as  many  as  you  wish.  The  default  is 
alphabetical.): 


□ Alphabetical 

□ By  Timeline 
n By  Location 


G By  People  Involved 
□ By  Batde  Event 


!g 

HI 


Back  to  Battle  of  Shiloh  Introduction  Page 
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Table  1.  Quiz  Score  Means  by  Battle  and  Instructional  Web  Type 


Instructional  Web  Type 


Battle 

Navigational 

Functional 

Adaptive 

Total 

Shiloh 

13.2" 

12.4 

16.7 

14.1 

Manassas 

14.3 

15.7 

16.5 

15.5 

Gettysburg 

15.8 

16.5 

17.4 

16.6 

Total 

14.4 

14.9 

16.9 

15.4 

Notes:  20  possible  correct  answers  per  battle.  "Each  battle  by  web  type  cell  reflects  mean  quiz  scores  for  4 students  in  2 dyads. 


Table  2.  Mean  Time  ( Minutes ) by  Battle  and  Instructional  Web  Type 


Battle 

- 

Instructional  Web  Type 

Navigational 

Functional 

Adaptive 

Total 

Shiloh 

52° 

43 

173 

89 

Manassas 

67 

82 

201 

116 

Gettysburg 

64 

77 

195 

112 

Total 

61 

67 

190 

106 

Notes:  each  battle  by  web  type  cell  reflects  the  time  for  2 dyads  each. 

Table  3.  Attitude  Response  Means  by  Web  Type 


Web  Type 


Survey  Item 

Navigational 

Functional 

Adaptive 

Learning  about  this  Civil  War  battle  was 
interesting. 

3.2® 

2.9 

3.7 

1 would  like  to  visit  other  Civil  War  websites  on 
the  Internet. 

3.3 

3.4 

3.6 

1 would  like  to  learn  more  about  the  Civil  War. 

3.4 

3.3 

3.5 

1 tried  hard  to  learn  the  information  presented 
about  this  battle. 

3.2 

3.5 

3.8 

I had  control  over  what  1 learned  related  to  the 
Civil  War  while  using  this  Civil  War  websites. 

3.1 

2.8 

3.8 

The  structure  of  this  website  encouraged  me  to 
explore  all  the  information  presented  about  my 
assigned  Civil  War  battle. 

3.6 

2.4 

3.6 

1 had  control  over  the  Civil  War  information 
presented  on  the  computer. 

3.0 

3.5 

3.8 

1 am  confident  that  1 will  do  well  on  a quiz  over 
this  battle. 

3.1 

2.8 

3.7 

Rate  the  level  of  interactivity  you  experienced 
while  learning  about  this  battle  from  the 
computer.b 

1.6 

2.2 

2.8 

Notes:  Each  cell  represents  12  students  (from  6 dyads)  responding.  “The  scale  for  these  items  was:  4 = strongly  agree,  3 = Agree,  2 = Disagree, 
1 = Strongly  Disagree,  ^e  rating  scale  for  this  item  was:  0 = Not  interactive  at  all,  1 = A little  interactive,  2 = Fairly  interactive,  3 = Very 
interactive. 
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Models  for  Web-Based  Instruction:  A Discussion  of  Recurring  Themes 


Craig  Edwards 
Northern  Illinois  University 


Abstract 

This  paper  will  review  some  of  the  current  themes  from  three  web-based  instruction  (WBI)  models.  These 

models  include:  Reeves  and  Reeves ' WBI  model , Ritchie  and  Hoffman's  model , and  the  DuchasteTs  model. 

Recurring  themes  from  these  three  models  will  be  consolidated  into  a model  under  development  for 

application  to  WBI  in  higher  education. 

Introduction 

Currently  there  are  no  time-proven  models  for  WBI  in  existence,  simply  because  the  Internet  has  not  been 
used  very  long  as  a means  of  instruction  in  higher  education.  The  author  contends  that  new  instructional  models, 
which  are  quite  different  from  the  traditional  instructional  models  found  in  today’s  universities  and  colleges,  must  be 
considered  and  various  aspects  and  strategies  must  be  adapted  from  existing  instructional  design  models.  These 
instructional  strategies  will  be  different  because  of  the  nature  of  teaching  over  the  Internet,  because  of  the  functions 
available  in  online  courses  and  because  of  the  lack  of  face-to-face  interaction  and  visual  cues.  The  Internet  allows 
for  new  methods  and  models  of  instruction,  most  of  which  could  and  should  employ  constructivist,  hands-on, 
learner-centered,  project-based,  collaborative  methods,  strategies  and  activities. 

Reeves  and  Reeves  WBI  Model 

This  model  includes  ten  areas  for  interactive  learning  on  the  WWW  called  dimensions.  They  emphasize 
that  what  is  important  about  the  web  is  not  the  media,  but  what  pedagogical  features  and  strategies  can  be  delivered 
via  WBI.  The  10  dimensions  they  propose  include  "(1)  pedagogical  philosophy,  (2)  learning  theory  (3)  goal 
orientation  (4)  task  orientation  (5)  source  of  motivation  (6)  teacher  role  (7)  metacognitive  support  (8)  collaborative 
learning  (9)  cultural  sensitivity  and  (10)  structural  flexibility"  (Reeves  and  Reeves,  1997,  p.  59).  Reeves  and  Reeves 
present  these  dimensions  as  two-ended  continuums  with  contrasting  concepts  at  each  end.  For  example,  they  define 
the  teachers'  roles  as  a continuum  between  didactic  and  facilitative  roles. 

Ritchie  and  Hoffman's  Model 

This  model  incorporates  instructional  design  principles  into  WBI.  Ritchie  and  Hoffman  (1997)  have 
classified  Dick  and  Rieser’s  (1989)  instructional  sequences  into  seven  events  that  can  be  incorporated  into  instruction 
for  the  World  Wide  Web.  These  elements  include:  motivating  the  learner,  explaining  what  is  to  be  learned,  helping 
the  learner  recall  previous  knowledge,  providing  instructional  material,  providing  guidance  and  feedback,  testing 
comprehension,  and  providing  enrichment  or  remediation. 

Duchastel's  Model 

Duchastel's  model  (1997)  proposes  radical  changes  to  the  concept  of  the  present  typical  university 
instruction  model.  His  web-based  model  for  university  instruction  proposes  several  functions  that  define 
advantageous  approaches  to  WBI.  These  functions  are  contrasted  with  the  usual  functions  found  in  traditional 
university  teaching.  They  form  a model  that  can  guide  the  development  of  courses  and  programs  that  are  more 
appropriate  to  WBI.  Professors  and  instructional  designers  could,  rather  than  specify  content  to  be  learned,  and 
common  learning  results,  specify  the  goals  to  be  pursued  and  accept  a variety  of  learning  outcomes.  Duchastel 
(1997)  suggests  evaluating  learners  at  the  task  level  rather  than  relying  on  the  standard  mode  of  testing.  And  instead 
of  having  learners  study  and  work  individually,  cooperative  and  collaborative  learning  teams  should  be  emphasized. 
Instructors  should  encourage  learners  to  produce  knowledge  rather  than  require  the  communication  of  knowledge. 
Lastly,  he  advocates  encouraging  global  learning  communities  rather  then  restricting  programs  to  local  interactions 
(Duchastel,  1997). 

Recurring  Themes 

At  first  glance,  it  might  seem  that  there  is  not  much  in  common  between  these  three  models.  There  are 
however,  several  recurring  themes.  The  first  is  the  need  for  clear  goals.  The  second  theme  suggests  that  WBI 
include  collaborative  learning  teams.  The  third  theme  addresses  the  issue  of  consciously  incorporating  motivational 
aspects  into  WBI.  The  forth  theme  revolves  around  the  idea  that  trainers  or  instructors  should  act  more  as 
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facilitators  or  coaches  who  provide  guidance  and  clear  feedback.  Finally,  the  fifth  theme  suggests  that  WBI 
facilitate  the  production  of  knowledge  and  the  development  of  skills. 

These  themes  form  the  basis  of  a working  model  of  WBI  in  higher  education  that  I am  in  the  process  of 
developing.  This  model  is  aimed  specifically  at  higher  education  and  adult  learners;  however,  elements  of  this  model 
could  certainly  be  applicable  to  any  type  of  WBI.  These  will  lay  the  foundation  for  issues  to  be  considered  when 
designing,  developing  and  delivering  WBI  for  higher  education. 

The  following  are  the  basic  foundations  of  this  web-based  instruction  model: 

• Goal-Oriented 

• Motivational  Provisions 

• Student-Centered 

• Guidance/Feedback  Provisions 

• Collaborative  Learning  Strategies 

• Project-Based 


Clear  Goals 

First,  WBI  should  provide  clear  goals  and  expectations.  Students  need  to  know  early  on  what  will  be 
expected  from  them  in  a WBI  course.  Instructors  should  have  a clear,  week  by  week  or  unit  by  unit  explanation  of 
the  activities  and  exercises  in  the  course.  In  WBI,  learners  are  often  unsure  of  how  to  proceed,  and  easily  lose  sight 
of  the  purpose  of  the  course.  Reiterating  the  goals  and  expectations  at  various  points  in  the  course  may  keep 
students  on  track. 

Student-Centered 

WBI  should  be  more  student-centered  with  the  instructor  as  a facilitator  or  coach.  Learners  should  take 
more  responsibility  for  recognizing  patterns  of  information,  organizing  the  information,  constructing  their  own  views 
and  perspectives  and  creating  and  presenting  knowledge  in  new  ways  (Reeves  & Reeves,  1997).  In  WBI,  learners 
can  and  should  be  presented  with  problems  to  solve,  and  projects  and  papers  to  complete.  In  WBI  the  tools  are 
readily  available  to  allow  the  instructor  or  trainer  to  take  on  the  role  of  facilitator,  coach,  mentor  or  guide. 

Guidance/Feedback  Provisions 

Since  there  may  be  little  or  no  face-to-face  interaction  in  an  online  course  students  will  require  a large 
amount  of  guidance  and  feedback.  This  can  be  accomplished  by  providing  the  students  with  a detailed  syllabus  and 
a clear  timeline  for  discussion  of  content,  activities,  projects  and  discussions.  It  is  also  important  that  students 
receive  feedback  in  a timely  manner.  E-mail  should  be  answered  within  a day  or  two.  If  there  is  no  time  for  a 
detailed  answer,  the  instructor  should  at  least  acknowledge  that  the  e-mail  was  received  and  a more  complete  answer 
will  be  sent  later.  This  answer  could  be  in  the  form  of  a posting  to  a FAQ  or  even  a group  e-mail  if  required.  In 
order  to  keep  students  on  track  and  informed,  the  instructor  may  want  to  consider  either  posting  a weekly  reminder  to 
the  course  web  site  or  sending  a group  e-mail.  Some  online  software  allows  for  a message  center  where  an  instructor 
can  post  messages  for  all  students  to  see.  An  online  journal  provides  a format  in  which  the  learner  record  reflections 
on  materials  covered  and  instructors  can  provide  personalized  guidance  and  feedback.  Finally,  it  is  important  the 
WBI  instructor  or  trainer  learn  strategies  for  facilitating  asynchronous  online  discussions  and  synchronous  chats. 
These  skills  come  over  time  with  practice  and  experience. 

Motivational  Provisions 

Motivational  issues  are  always  essential  in  instruction,  but  they  are  much  more  important  for  WBI  because 
there  is  no  face-to  face  interaction  available.  Learners  in  a WBI  environment  will  benefit  greatly  if  provided  with 
positive  feedback,  relevant  problem-based  learning  (PBL)  activities,  relevant  examples,  and  the  establishment  of  the 
value  of  skills  to  be  acquired  in  the  WBI  course  (Duchastel,  1997). 

Collaborative  Learning 

There  are  a rich  array  of  possible  techniques  and  strategies  for  incorporating  collaborative  and  cooperative 
learning  into  WBI  (Bonk  & Reynolds,  1997).  Students  could  collaborate  on  projects  via  e-mail,  or  through 
synchronous  or  asynchronous  conferencing.  Students  in  conferencing  situations  could  be  assigned  roles  such  as  a 
leader,  initiator  or  facilitator  of  discussion  topics  (Bonk,  Appelman,  & Hay,  1996).  Giving  students  roles  in 
discussions  facilitates  students'  processing  of  the  content.  Structured  controversy,  also  known  as  academic 
controversy,  Millis  & Cottell,  1998)  is  another  strategy  which  would  works  well  in  the  WWW  environment.  Teams 
of  students  would  be  assigned  opposing 

viewpoints  of  a controversial  topic.  The  teams  would  gather  materials  and  present  their  sides.  Later  teams  would 
reverse  their  roles  and  argue  for  the  opposing  viewpoint.  (Millis  & Cottell,  1998) 
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Project-Based 

The  web  offers  a rich  variety  of  possibilities  for  project-based  activities.  Students  can  collaborate  on 
research  papers,  on  problem/solution  activities,  authentic,  real-world  and  practical  activities.  The  hyperlinking 
capabilities  of  the  Web  and  the  large  amount  of  resources  online  also  present  many  possibilities  for  rich  project- 
based  activities.  These  types  of  activities  give  learners  the  chance  to  acquire  kills  useful  in  real-world  situations  and 
experience  working  as  a member  of  a team. 

Conclusion 

This  model  is  far  from  complete  and  is  not  intended  to  cover  every  single  aspect  of  WBI  in  higher 
education.  Rather  it  aims  to  give  instructional  designers,  course  developers  and  faculty  a starting  point  for  the  design 
and  development  of  WBI.  This  model  is  still  very  much  in  the  process  of  being  developed,  and  fine-tuned.  The 
author  invites  suggestions  and  feedback  on  this  Web-based  instructional  model. 

Craig  Edwards  E-mail  - crai g @ dragonco . com 
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Multiple  Instructional  Design  Models:  Necessity  or  Excess  Baggage? 


David  Beck 
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Abstract 


The  topic  of  the  proliferation  of  Instructional  Design  models  is  often  overlooked,  ignored  and  considered 
cliche.  This  paper  explores  the  necessity  for  the  multitude  of  models,  as  well  as  a simple  framework  for 
keeping  them  organized.  Additionally ; a rationale  is  given  for  accepting  empirically  validated  instructional 
design  models.  The  reasons  include  professional  accountability  and  historical  honesty  (we  learn  from  our 
past  trials  and  tribulations).  Furthermore,  the  author  posits  that  the  learning  process  and  the  design 
environment  are  far  too  complex,  and  our  knowledge  of  the  learning  process  incomplete,  to  accept  the 
status  quo  of  any  one  particular  model. 

A brief  reflection  of  the  past  30  years  of  Instructional  Technology  reveals  an  ever-evolving  profession. 

Deep  within  this  evolution  are  the  instructional  design  models  that  comprise  the  methods  by  which  designers  ply 
their  trade.  A review  of  the  literature  relating  10  this  period  of  time  supports  this  notion.  For  instance,  McCombs 
(1986)  posits  that  the  years  between  1970  and  1985  represent  a period  of  time  of  significant  change  within 
instructional  design  models.  However,  the  inherent  change  during  this  time  didn't  effect  the  rather  ubiquitous 
category  of  general  design  tasks.  It  was  the  instructional  design  strategies  and  processes  that  continued  to  change  in 
response  to  the  demands  of  the  times. 

In  the  late  1990's,  Instructional  Technology  can  still  be  considered  a relatively  new  discipline  (Sullivan  et 
al,  1997).  Professions  yet  to  mature  can  expect  to  undergo  periods  of  significant  change  and  even  evolution.  The 
status  of  instructional  design  models  is  positioned  at  the  forefront  of  today's  contribution  to  this  evolution.  In  a 
context  similar  to  the  one  asserted  by  McCombs  (1986),  Willis  (1998)  suggests  “that  models  continue  to  change  as 
new  theories  of  learning  and  instruction  become  popular  and  influential”  (p.5). 

These  changes  are  anchored  by  a desire  to  understand  how  learning  occurs  and  how  to  best  facilitate  the 
occurrence  of  learning.  Furthermore,  Merrill  (1991)  postulates  that  very  little  is  known  about  the  acquisition  of 
knowledge.  It  would  appear  that  Instructional  Design  will  continue  to  evolve  as  Instructional  Technology 
professionals  strive  to  understand  the  learning  process  and  utilize  their  findings  to  augment  existing  design 
methodologies  and  develop  new  paradigms. 

Just  as  instructional  design  models  continue  to  evolve,  so  have  the  number  of  models  increased.  Edmonds 
(1994)  found  that  “since  the  first  appearance  of  instructional  design  models  in  the  sixties,  there  has  been  an  ever 
increasing  number  appearing  in  the  literature  of  both  instructional  technology  and  general  education”  (p.55). 
Additionally,  Gustafson  and  Branch  (1997)  acknowledge  the  amount  of  research,  and  the  number  of  related  models, 
has  grown  disproportionately  through  the  expanded  use  of  systematically  designed  instruction.  Consequently,  a 
systematic  approach  dominates  as  the  prevailing  instructional  design  methodology. 

Related  to  the  systematic  design  of  instruction  is  Instructional  Systems  Design  (ISD)  of  which  there  are 
many  representative  models.  It  has  been  suggested  that  ISD  provides  those  directed  with  designing,  developing,  and 
delivering  instruction  an  excellent  means  of  controlling  quality,  providing  a common  source  of  communication,  and 
managing  projects.  Additionally,  McCombs  (1986)  found  that  ISD  models  generally  address  all  design  tasks  and, 
because  of  their  methodic  nature,  lend  themselves  well  to  virtually  any  instructional  design  project.” 

Alternatives  to  the  traditional  ISD  models  are  becoming  more  prevalent.  At  the  forefront  of  this  charge  are 
influences  based  upon  Constructivist  tenets  (Willis,  1995)  and  the  fields  of  performance  technology  and 
organizational  development(Reigeluth,  1996).  These  tenets  are  impacting  the  way  traditional  instructional  design 
models  are  evolving  (Dick,  1996;  Reigeluth,  1996)  and  new  models  are  developing.  Trying  to  decipher  the  nuances 
of  each  and  every  model  presents  an  imposing  task  for  researchers  and  practitioners  alike. 

Comparative  Frameworks 

To  help  with  this  task,  several  individuals  have  proposed  a taxonomy  of  ID  models  based  upon  grouping  of 
like  characteristics.  Edmonds  (1994)  suggests  that  Andrews  and  Goodson  (1980)  and  Gropper  (1983)  were  primarily 
responsible  for  early  efforts  in  this  area.  Andrews  and  Goodson’ s (1980)  framework  was  comprised  of  categories  or 
dimensions:  Origin,  Theoretical  Underpinnings,  Purposes  and  Uses,  and  Documentation.  Gropper  (1983)  presented  a 
metatheory,  which  is  intended  to  compare  instructional  design  theories  and  models.  A work  by  Gustafson  and  Branch 
(1997b)  suggests  that  instructional  design  models  could  be  placed  into  one  of  three  categories:  classroom  focused, 
product  focused,  and  system  focused.  Edmonds  (1994)  builds  upon  the  works  of  others  by  extending  his  analysis  to  a 
framework  that  includes  context,  orientation,  knowledge  structure,  expertise  level,  structure,  and  level.  This 
framework  was  then  developed  into  a matrix  as  a means  of  simply  comparing  instructional  design  models.  Other 
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frameworks  are  either  aimed  at  suggesting  the  superiority  of  one  model  over  another  or  aren’t  comprehensive 
enough. 

The  frameworks  cited  are  very  appropriate  for  their  intended  purpose.  With  the  exception  of  the  work  by 
Gustafson  and  Branch  (1997b),  however,  the  frameworks  aren’t  very  intuitive  for  individuals  with  less  than  an 
advanced  knowledge  of  instructional  design  terminology.  This  creates  a problem  of  interpretation.  Methodological 
sophistication  is  not  worth  much  if  you  cannot  reach  the  audience  you  wish  to  reach,  and  it  is  difficult  to  serve  both  a 
technical  audience  and  a general  audience  in  the  same  work.  The  framework  represented  in  Table  1 is  a modification 
of  the  frameworks  listed  above  (see  Andrews  & Goodson,  1980;  Edmonds  et  al.,  1994;  Gustafson  & Branch,  1997b). 
Its  purpose  is  to  simplify  the  comparison  of  models  and  reach  an  audience  far  less  technical  in  orientation  than  those 
served  by  more  sophisticated  frameworks. 

Table  1 Modified  Comparative  Framework 


K:>K  Ms 

Expartfw 

:'?•<  Product  •' 

Orientation 

Knowledge  Structured 

tfisit 

Unit, 

Course, 

Descriptive 

Prescriptive 

Procedure 1 

Declarative 

Module, 

Institution 

■V 

Lesson 

ID! 

Novice* 

X 

X 

X 

Gertach  and  Ely 

Novice 

X 

X 

X 

Dick  and  Reiser 

Novice 

X 

X 

X 

X 

Heinich/Assure 

Novice 

X 

X 

X 

X 

R2D2 

Expert 

X 

X 

X 

Diamond 

Expert 

X 

X 

X 

X 

X 

Kemp 

Novice 

X 

X 

X 

X 

Gerlach  and  Ely 

Novice 

X 

X 

X 
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X 

X 

X 

X 

Chaos 

Expert 

X 

X 

X 

X 

Dick  and  Carey 

Novice 

X 

X 

X 

X 

IPISD 

Novice 

X 

X 

X 

X 

Van  Patten 

Expert 

X 

X 

X 

Bergman  and  Moore 

Novice 

X 

X 

X 

X 

TrainTri^^'^ 

Kemp 

Novice 

X 

X 

X 

X 

Van  Patten 

Expert 

X 

X 

X 

Dick  and  Carey 

Novice 

X 

X 

X 

X 

. . • «i.^>  iv. 

Rapid  Prototyping 

Expert 

X 

X 

X 

X 

Although  most  of  the  general  classifications  were  retained  from  earlier  works  (see  Edmonds,  Branch  et  al., 
1994),  the  categories  were  rearranged  for  ease  of  reading.  The  classifications  with  brief  definitions  are  as  follows: 

• Context.  Similar  to  Gustafson’ s(  1998)  framework.  The  type  of  environment  for  which  an  ID  model  appears  best  suited 
first  delineates  table  1. 

• ID  Model.  A representation  of  systems  and  non-systems  based  ID  models. 

• Expertise.  The  level  of  knowledge  one  must  possess  before  attempting  to  utilize  the  model. 

• Product.  The  type  of  product  for  which  the  model  appears  best  suited. 

• Orientation.  Whether  a particular  model  is  better  at  prescribing  or  describing  the  steps  of  the  ID  process. 

• Knowledge  Structure.  Procedural  or  declarative  related  outcomes.  The  procedural  aspect  relates  to  how  learners  reach  their 
goals.  With  the  declarative  orientation,  the  learner  is  able  to  state  why  something  is  the  case.  (Edmonds  et  al,  pp.60-7 1 ) 


It  is  important  to  note  the  various  categories  aren’t  mutually  exclusive.  Just  because  a model  is  considered 
novice,  doesn’t  mean  that  it  isn’t  well  suited  for  expert  level  designers.  On  the  other  hand,  expert  level  models  are 
more  or  less  heuristic  and  probably  wouldn’t  be  well  suited  for  novice  designers.  Similarly,  some  models  may  not  be 
adequate  for  more  than  one  context  area  (see  Edmonds  et  al,  1994). 

Conclusion 

Models  are  a vehicle  of  communication  ...  a way  of  making  abstract  ideas  concrete.  Additionally,  models 
can  serve  as  algorithms  for  problem  solving,  as  avenues  for  scientific  discovery,  and  as  job  aids.  Instructional 
Technologists  have  done  well  to  provide  their  peers  with  a plethora  of  models  from  which  to  choose.  The  framework 
presented  in  this  paper  is  one  of  the  ways  in  which  these  models  may  be  organized.  The  models  this  paper  represents 
are  only  a few  of  the  multitude  that  exist.  Are  all  of  these  models  necessary?  The  answer  is  a qualified,  it  depends. 

Much  like  an  architect,  an  instructional  designer  is  able  to  better  construct  meaningful  relationships  with  his 
or  her  environment  once  the  foundational  relationships  are  understood.  Whether  the  model  of  choice  at  the  time  is 
the  Dick  and  Carey  model  or  a model  by  any  other  name,  awareness  of  the  other  design  models  within  the  profession 
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has  its  merits.  To  not  consider  the  merits  of  other  models  is  shortsighted  and  professionally  irresponsible.  Is  it 
responsible  to  accept  the  status  quo  of  any  particular  model,  when  we  don’t  completely  understand  the  learning 
process  and,  therefore,  the  best  way  to  facilitate  the  learning  process?  Reigeluth  (1996)  asserts,  that  “we  should  not 
reject  the  basic  tenets  of  current  and  past  ISD  models.  These  paradigms  offer  the  knowledge  generated  from  past 
experiences  and  provide  us  with  the  building  blocks  for  future  direction.” 

There  is  a problem,  however,  when  these  paradigms  or  models  lack  a firm  foundation  upon  which  to  build 
this  future.  Many  of  these  models  lack  the  scientific  proof  necessary  to  provide  evidence  of  validity.  Willis  (1998) 
states,  that  “we  can  adopt  a behavioral,  a cognitive,  a critical,  or  a constructivist  framework  for  design,  but  whatever 
framework  we  adopt,  it  will  be  on  the  basis  of  something  other  than  convincing  body  of  research  that  demonstrates 
that  framework  is  stronger  than  the  others”  (p.  12). 

It  becomes  incumbent  of  all  stakeholders  to  critically  assess  the  validity  of  instructional  design  models.  On 
one  hand,  there  is  a need  for  researchers  to  continue  investigations  that  help  bridge  the  gap  between  theory  and 
practice.  Consequently,  Developmental  research  can  show  how  models  and  theories  work  and  identify  situations  of 
optimal  performance.  Richey  (1997)  On  the  other  hand,  practitioners  can  contribute  by  staying  abreast  of 
professional  developments.  By  doing  so,  there  is  an  increased  likelihood  that  models  are  utilized  as  intended  and 
learning  strategies  match  the  desired  outcomes. 

Comparative  frameworks,  such  as  those  proposed  by  Gropper  (1983),  Andrews  and  Goodson  (1980),  and 
Gustafson  (1997b),  provide  a means  by  which  many  of  the  models  may  be  compared  and  organized.  The  author’s 
modified  framework  is  meant  as  a tool  for  a novice;  a starting  point  which  is  seen  as  less  confusing  (and  rigorously 
developed)  than  many  of  the  other  frameworks  discussed  within  this  paper.  As  models  are  compared  and  either  gain 
acceptance,  lose  favor,  or  are  forged  into  new  paradigms,  both  learner’s  and  Instructional  Technology  wins.  This 
means  the  likelihood  increases  that  individuals  eager  to  learn  will  be  able  to  do  so  in  an  environment  conducive  to 
the  pursuit  of  their  endeavors,  with  materials  and  media  designed  to  optimize  their  learning  experience  and  abilities. 
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This  paper  reports  the  findings  of  an  ongoing  study  of  engagement  in  computer-based  learning 
environments.  Participants  engaged  in  both  open  systems  and  closed  systems  were  studied  to  ferret  out 
strategies  of  learner  engagement.  Findings  indicate  that  problem  identification , physical  representation, 
and  interaction  styles  can  be  manipulated  to  help  foster  engagement  of  learners  in  computer-based 
learning  environments. 

Introduction 

The  computer  and  all  of  its  digital  manifestations  are  causing  us  to  rethink  the  way  we  do  everything  from 
producing  to  disseminating  media  (Negroponte,  1996).  The  term  Virtual  Reality  (VR)  is  meant  to  imply  that  the 
reality  exists  outside  of  the  physical  world.  For  a number  of  people,  computer  generated  reality  isn’t  virtual  at  all. 
Computer  generated  environments  are  real:  they  just  happen  to  be  on  the  computer.  This  can  best  be  seen  in 
computer  and  video  games,  where  people  may  spend  their  time,  to  the  point  of  distraction,  playing  these  games.  It 
matters  not  that  the  environment  has  no  counterpart  in  the  physical  world.  What  matters  is  the  quality  of  the 
interactions,  the  problems,  and  representations  of  them.  People  not  only  play  in  these  environments,  but  they  live 
within  these  environments  as  well  (Herz,  1997).  For  better  or  worse,  digitally  rendered  environments  can  be  as,  at 
times  more,  engaging  than  reality.  While  this  is  true  for  computer-based  gaming  environments,  this  is  not  true, 
necessarily,  for  computer-based  learning  environments  (Jones,  1997).  The  impetus  for  this  study  was  not  that 
educational  software  should  look  like,  or  in  any  way  mirror  computer  games.  Educational  software  that  tries  to  copy 
gaming  patterns  tends  to  be  fatuous.  Rather  the  question  was  are  there  engagement  strategies  that  are  used  in  games 
that  can  be  applied  to  educational  software.  Again,  the  desire  was  to  look  at  strategies  and  not  techniques.  The 
purpose  of  this  study  was  to  look  at  what  engages  people  in  computer  games,  and  to  see  how  those  patterns  of 
engagement  might  be  used  within  computer-based  learning  environments.  The  findings  presented  here  represent 
nearly  two  years  of  studying  various  populations  of  people  as  they  play  games  and  work  in  computer  based  learning 
environments.  The  study  has  progressed  to  include  both  open  and  closed  environments  (Jones  & Farquhar,  1997). 
The  ages  of  participants  ranged  from  age  4 to  adulthood.  The  number  of  study  participants  to  date  is  17.  However  it 
should  be  noted  that  each  individual  may  have  been  engaged  and  studied  in  multiple  environments.  Current  research 
is  looking  at  on-line  formal  and  informal  learning  communities. 

Procedures 

Engagement  here  is  defined  as  “...nexus  of  intrinsic  knowledge  and  or  interest  and  external  stimuli  that 
promote  the  initial  interest  in,  and  continued  use  of  a computer-based  learning  environment”  (Jones,  1998,  pg.  205). 
Engagement  was  studied  in  three  ways:  (1)  direct  observations  of  people  as  they  worked  in  an  environment;  (2) 
interviews  with  people  after  they  had  been  engaged  in  the  environment;  (3)  analysis  of  written  discourse  through  e- 
mail,  journals,  and  other  artifacts.  Many  of  the  interviews  were  conducted  at  a distance,  consequently  not  all 
participants  were  observed  working  in  the  environment. 

Findings 

Findings  of  the  study  are  presented  through  a combination  of  narratives  and  static  graphics.  Because  many 
of  the  findings  discussed  here  involve  intricate  interactions,  it  may  be  difficult  to  represent  them  on  a printed  page. 

Problem  Definition 

Engagement  was  found  to  be  enhanced  due  to  the  nature  of  the  problem  itself.  Problems  that  are  accepted 
by  a user  can  promote  engagement,  and  consequently  the  user  maintains  interest  in  the  problem.  Interest  in  the 
problem,  or  having  a well  defined  question,  provides  the  learner  with  intrinsic  motivation  to  be  working  within  the 
environment. 

Learners  who  are  intrinsically,  or  self,  motivated  have  found  that  something  as  generic  as  a multi-media 
encyclopedia  can  be  engaging.  The  motivation  comes  from  within,  and  the  environment  itself  need  not  be  engaging 
beyond  its  ability  to  present  the  content  in  a factual  manner.  Learners  have  reported  being  engaged  in  generic 
environments  such  as  data  bases.  For  example,  one  participant  was  relocating  to  another  city,  and  was  doing 
research  on  their  new  city.  While  not  formal  didactic  instruction,  the  activity  was  certainly  about  learning. 
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Ned  and  his  family  were  relocating  to  Tampa,  FL  from  Chicago,  IL.  Among  the  many  things  he  had  to  do  was  to 
find  a house.  Ned  is  a 60  year  old  college  professor.  While  he  has  access  to  the  WWW,  he  has  never  used  it 
beyond  very  simple  browsing.  When  shown  a web  site  listing  real  estate  for  sale  in  Tampa,  his  interest  and 
expertise  increased  as  his  use  increased.  The  problem  itself  provided  him  with  a compelling  reason  to  be  on  line. 

He  reported  increased  interest  in  the  web,  and  demonstrated  greater  command  of  the  online  world.  “It  is  funny.  I 
started  out  looking  for  a house,  but  what  I found  was  a collection  of  information  that  was  available  to  me.  I soon 
found  out  that  I could  manage  one  of  retirement  accounts  on  line.  I never  thought  that  I would  ever  do  this.” 

The  point  is  not  a dramatic  conversion  to  the  WWW.  The  point  is  that  Ned  found  a problem  that  engaged 
him,  thus  keeping  him  actively  engaged  in  the  environment.  Finding  other  problems  that  were  equally  engaging  has 
kept  him  using  the  web.  Ned  has  not,  at  this  reporting,  become  an  expert  HTML  programmer  and  opened  an  e- 
commerce  operation.  It  is  unlikely  that  he  will.  He  will,  however,  continue  to  use  the  WWW  to  solve  certain  types 
of  problems  thus  pointing  further  that  problem  identification  by  the  user  can  be  an  important  means  of  engaging  the 
user.  Learners  who  can  generate  their  own  problems  often  demonstrate  greater  manifestations  of  engagement  in 
open  systems.  In  closed  systems,  the  problem  is  often  generated  for  the  learner. 

If  the  learner  is  not  intrinsically  motivated,  or  interested  in  the  problem  the  environment  provides,  then  the 
environment  may  need  to  offer  greater  extrinsic  motivational  features  to  keep  the  learner  interested.  This  was  noted 
in  non-golfers  playing  computer-based  golf  games.  People  who  were  not  golf  players  in  the  physical  world  were 
often  engaged  in  golf  games.  There  are  many  reasons  for  this.  It  is  less  time  consuming  to  play  golf  on  a computer. 
It  is  less  expensive;  less  stressful,  and  for  many  people,  less  embarrassing.  One  of  the  key  reasons  stated  by  several 
participants  was  that  the  manner  in  which  the  game  was  represented  gave  them  the  impetus  to  play  it. 

“I  don’t  really  have  the  time  to  play  golf.  1 have  always  thought  it  to  be  kind  of  an  elitist  activity.  And  I really 
don’t  think  I want  to  play  golf  ‘live.’  But  I did  think  that  the  golf  game  was  kind  of  cool.  It  didn’t  feel  much  like 
golf,  but  more  like  just,  you  know,  playing  any  other  kind  of  game.  ...the  music  in  the  background  was  a little 
weird,  but  the  graphics  were  very  good.  I liked  having  the  control  over  the  game.  Choosing  the  clubs,  having  to 
click  just  right  to  get  the  right  swing. . . .” 

While  this  person  did  not  find  golf  that  interesting,  the  environment  about  golf  was  interesting.  And  while 
problems  which  are  fatuous  or  uninteresting  do  not  promote  engagement  regardless  of  the  quality  of  the  visuals  or 
other  features  of  the  product,  the  quality  of  the  presentation  appears  to  have  an  impact  on  whether  or  not  somebody 
will  try  the  problem.  This  leads  into  the  next  area,  visual  quality. 

Visual  quality 

The  gaming  community  appreciates  not  only  good  problems,  but  well  rendered  problems  as  well.  In  order  to 
contribute  to  the  engagement  of  the  user,  graphics  and  other  multi-media  assets  must  not  only  be  well  rendered,  but 
must  also  contribute  to  the  gestalt  of  the  environment.  Including  graphics  as  “window  dressings”  is  not  appreciated, 
and  can  impact  engagement  levels  negatively.  As  one  player  in  a computer  game  reported: 

“I  don’t  know  why  they  included  those  video  clips  on  top  of  the  heads  of  the  still  graphics.  I suppose  it  is  because 
they  could.  But  honestly  I started  to  think  of  some  old  Saturday  Night  Live  gag,  and  I couldn’t  take  it  seriously.” 

In  looking  at  the  use  of  multi-media  assets  in  varying  environments,  it  has  become  useful  to  look  at  them 
through  the  lens  of  Csikszentmihalyi’s  (1990)  Flow  Theory.  I propose  that  there  are  two  types  of  aesthetic  elements, 
active  and  passive  aesthetics  (Jones  & Surry,  In  Press). 

Passive  Aesthetics 

Passive  aesthetics  may  be  aligned  with  Csikszentmihalyi’s  (1990)  definition  of  pleasure.  Pleasure  is  defined 
here  as  being  passive  in  nature,  and  not  requiring  active  participation  by  an  individual.  Passive  aesthetics  are 
elements  in  the  program  that  may  provide  visual  or  auditory  texture  to  the  program.  For  example,  background 
images  in  a program  can  do  a number  of  things.  A background  can  provide  depth  and  texture  to  your  program.  It 
can  provide  learners  with  a visually  pleasing  image  that  is  not  overwhelming  to  look  at,  and  which  may  provide  the 
developer  with  an  interesting  canvas  on  which  to  “paint.”  Additionally,  background  music  can  help  set  the  tone  for 
an  environment  as  well.  For  example,  the  driving  beat  in  Doom  II  provides  a feeling  of  intensity,  while  the  nature 
sounds  in  Golf  give  one  the  feeling  of  being  out  of  doors.  Background  images  may  help  the  user  maintain  the  feeling 
of  belief  in  the  environment.  Simply  having  an  attractive  background  is  not  enough.  The  background  must  sell  the 
user  on  the  underlying  theme  (I  hate  to  say  metaphor)  of  the  program. 

In  educational  software,  passive  aesthetics  are  often  attended  to.  It  is  not  difficult  to  locate  background 
images  that  can  add  visual  depth  and  texture  to  the  program.  The  washed  out  image  residing  behind  control  bars  is 
quite  familiar  to  us.  It  is  seen  in  both  educational  CD-ROMS  (closed  systems)  and  web-based  environments  (open 
systems).  What  is  more  difficult  is  finding  tools  that  are  integrated  into  the  visual  system  of  the  program  which 
perform  the  function  intended.  We  refer  to  these  elements  are  active  aesthetics. 
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Active  Aesthetics 

It  is  easier  to  locate  active  aesthetics  in  commercial  computer  games  than  in  educational  software.  It  is  also 
difficult  to  represent  them  on  the  printed  page.  Active  aesthetics  are  tools  within  the  environment  which  perform 
particular  functions  within  the  environment.  For  example,  in  PGA  Tour  Golf  III  your  goal  is  to  manage  all  the  data 
available  to  play  the  hole  well.  The  user  has  access  to  wind  direction  and  speed,  the  type  of  terrain,  and  suggested 
club  to  be  used.  This  is  represented  in  Figure  2.  These  pieces  of  the  program  all  work  together  to  weave  a seamless 
fabric  of  integration  of  images,  content,  and  control.  It  works  to  provide  the  user  with  an  environment  that  works  to 
promote  the  engagement  of  the  user. 

Figure  2.  Tools  Working  Together  In  An  Environment 


Within  educational  programs,  active  aesthetics  serve  as  tools  to  help  people  interact  with  the  software.  At 
its  most  basic  level,  we  are  talking  about  stimulus  and  response  activities  here,  and  stimulus  response  activities  are 
often  pejoratively  associated  with  behavioral  psychology  and  basic  drill  and  practice  activities.  But  it  is  important  to 
note  that  what  involves  people  in  a computer  based  environment  is  being  involved.  Users  should  be  clicking, 
dragging,  calculating,  comparing,  and  generally  interacting  with  the  information  in  the  program  itself.  Simply  being 
presented  with  the  information  is  not  enough  for  many  learners.  Working  with  the  information  takes  on  paramount 
importance.  It  follows  that  if  we  are  to  develop  interactive  materials  then  we  need  to  develop  methods  to  allow  users 
to  interact.  Computer  games  provide  two  types  of  interaction  to  examine. 

Types  of  Interactions 

Interaction  in  computer  games  can  be  classified  in  two  ways:  Twitch  and  Strategy  games.  Strategy  games 
are  ones  in  which  the  user  must  employ  higher  order  thinking  skills  and  problem  solving  skills  to  continue  playing 
and  win  the  game.  Twitch  games  are  games  in  which  the  user  must  react  quickly  to  circumstances,  usually  by  killing 
someone,  to  continue  playing  and  win  the  game  (see  Figure  3).  SimCity  and  War  Craft  II  are  immediate  examples  of 
strategy  games,  while  games  like  Doom  and  its  various  representations  are  consummate  twitch  games.  The 
advantage  to  a twitch  game  is  that  the  movement  is  quick,  and  the  feedback  immediate.  This  works  to  keep  the  user 
actively  engaged.  However  the  level  of  this  engagement  is  often  superficial.  It  does  not  typically  engage  one  beyond 
the  most  basic  level  of  seeing,  pointing,  and  clicking.  Strategy  games  require  the  user  to  look  at  the  larger  problem, 
and  plan  a strategy  to  solve  the  problem.  In  some  games,  such  as  SimCity , the  results  of  your  decisions  are  not 
immediately  recognized.  You  must  have  a fair  amount  of  internal  motivation  to  stay  with  the  game  to  realize  the 
fruits  of  your  labor.  While  twitch  games  offer  immediate  results  of  your  work,  strategy  games  appear  to  offer  a 
greater  feeling  of  accomplishment  and  satisfaction.  One  participant  who  was  playing  War  Craft  praised  the 
combination  of  “twitch  and  strategy.”  While  they  ultimately  liked  working  on  complex  problems  in  an  environment, 
they  also  appreciated  the  sheer  visceral  rush  of  immediate  feedback. 

Consequently,  when  designing  learning  environments,  it  should  be  the  goal  to  include  ‘last  action”  along 
with  a more  unifying  problem  to  be  solved.  One  simple  method  of  doing  this  is  to  include  twitch  type  mini-games 
within  a larger  strategy  game.  Monty  Python's  Quest  for  the  Holy  Grail  does  this,  as  does  IIth  Hour.  And  while  it  is 
not  seamless,  and  can  appear  a bit  crude,  it  does  at  least  provide  a faster  element  to  the  environment  than  many 
strategy  games  typically  afford.  Educational  games  such  as  Math  Blaster  offer  the  twitch  with  some  elements  of 
thinking,  but  these  types  of  games  are  simple  drill  and  practice  activities  in  disguise.  They  are  good  for  practicing 
certain  skill  sets,  but  do  little  to  help  develop  higher  order  thinking  skills.  One  possibility  would  be  to  use  math 
skills  to  solve  certain  problems  that  relate  to  the  larger  problem.  The  difference  is  that  the  individual  skills  then 
become  tools  to  solving  the  larger  problems,  which  is  ultimately  the  goal  of  education  in  the  first  place. 
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Point  of  View 

Engagement  can  be  done  through  how  the  user  sees  or  controls  the  screen.  Referred  to  as  Point  of  View 
(POV),  the  reference  point  that  one  takes  can  influence  how  one  is  engaged.  Looking  at  the  screen  down  the  barrel 
of  a gun  (see  Figure  3)  engages  you  in  the  world  as  the  participant.  You  never  see  the  entity  being  controlled:  it  is 
assumed  you  are  the  entity  and  you  are  controlling  yourself. 

Figure  3.  A Single  Point  Of  View:  Looking  Down  Barrel  Of  A Gun  In  A Twitch  Game 


Other  games,  such  as  Golf,  (see  Figure  2)  have  you  see  the  person  you  are  controlling.  There  is  a distinct 
visual  difference  between  the  two.  In  one,  you  are  looking  at  the  world  as  a participant.  In  the  other,  you  are 
looking  at  the  world  as  a disconnected  observer/controller.  People  engaged  in  these  environments  do  not  report  a 
difference  between  which  environment  is  likely  to  be  more  engaging  initially.  Nor  does  there  appear  to  be  any 
difference  in  a person’s  ability  or  likely  hood  to  accept  that  the  entity  being  controlled  is  the  user  in  either 
environment.  The  key  difference  appears  to  be  that  over  time  many  people  become  bored  with  a single  point  of 
view.  Therefore  it  may  follow  that  multiple  perspectives  in  the  environment  can  help  keep  the  user  engaged  in  the 
environment.  Working  from  a single  POV  can  promote  tunnel  vision,  and  may  not  support  continued  use  over  time. 
This  supports  something  that  has  been  known  in  the  gaming  community  for  a number  of  year:  A level  is  still  just  a 
level.  The  quality  of  the  level,  the  problems  one  encounters,  how  those  problems  are  manifested,  will  determine  how 
well  it  is  received  by  the  user.  Many  games  provide  the  user  with  multiple  points  of  view.  One  may  be  from  the 
perspective  of  your  character  or  agent,  the  other  may  be  a map  view  to  help  you  navigate  a maze,  as  in  Doom  //,  the 
other  may  be  a birds  eye  view,  as  in  many  golf  games,  to  let  you  see  the  entire  hole  at  one  time.  Multiple  views 
provide  multiple  perspectives,  or  multiple  representations  of  the  phenomenon.  Multiple  perspective  appear  to 
provide  a greater  feeling  of  engagement  on  the  part  of  the  user.  In  Kyoto:  The  Cosmology  of  Old  Japan , the  user  has 
the  opportunity  to  determine  physically  how  they  are  represented  (See  Figure  4).  There  appears  to  be  some  support 
for  allowing  this  kind  of  control  over  the  POV. 

In  Kyoto:  The  Cosmology  of  Old  Japan  the  user  has  the  option  of  selecting  the  features  of  their  character  in  the 
program.  One  can  select  gender,  body  type , and  facial  features . Among  participants ; this  was  found  to  be  a 
unique  feature,  Only  one  person  represented  themselves  as  a different  gender \ but  many  people  played  the  game 
several  times  from  different  perspectives.  Sue , an  18  year  old  college  freshman  said:  “ What  was  weird  was  that 
I felt  like  different  things  happened  because  of  the  shape  of  my  face . / don  7 know  if  they  did,  or  if  my  awareness 
of  it  changed.  ...I  definitely  knew  that  / was  different  in  the  game.  " 
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Figure  4.  Changing  The  Face  Of  Your  Character  In  An  Environment 


Future  Research 

In  terms  of  problem  identification  for  users,  this  appears  to  provide  greater  support  for  student  centered 
learning  environments.  As  we  move  from  more  traditional  closed  learning  environments,  such  as  educational  CD- 
ROMS  to  more  open  environments  as  can  be  found  on  the  WWW,  the  notion  of  letting  learners  generate  the  problem 
becomes  more  critical.  The  one  area  that  those  of  us  working  in  educational  software  need  to  consider  more 
carefully  is  representation  of  our  materials.  It  is  becoming  increasingly  important  to  produce  materials  that  can 
challenge  a sophisticated  group  of  users.  As  always,  more  research  is  needed  to  fully  understand  exactly  how  and 
why  people  are  engaged  in  computer-based  learning  environments. 
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Abstract 

The  purpose  of  this  study  was  to  assess  the  effectiveness  of  two  generative  learning  strategies,  concept 
maps  and  manipulation  of  objects,  and  to  determine  if  either  works  better  with  individual  learners  or  in 
cooperative  learning  groups.  A total  of  80  sixth-grade  students  in  science  education  were  randomly 
assigned  to  group  or  individual  conditions  and  to  one  of  the  two  experimental  treatments.  Experimental 
treatments  were  changed  between  first  and  second  post-test.  Long-term  retention  was  evaluated  with  a 
third  and  delayed  post-test.  Students  starting  with  concept  maps  showed  higher  achievement  on  delayed 
post-test  than  students  beginning  with  manipulation  of  objects.  No  difference  was  found  between  students 
working  as  individuals  or  cooperative  student  teams.  Furthermore,  a significant  interaction  between 
generative  learning  strategy  and  grouping  condition  was  revealed.  Implications  for  sequencing  generative 
learning  strategies  are  discussed. 

Introduction 

uHow  can  I help  my  student  learn?” 

The  goal  of  increased  student  learning  is  a fundamental  concern  for  teachers.  Over  the  years,  learning 
techniques  and  strategies  have  evolved  as  our  understanding  of  learners,  and  how  they  acquire  knowledge,  has 
grown.  From  the  1950s  to  the  early  1980s,  much  of  the  prevailing  educational  research  focused  on  improving 
teaching  by  analyzing  elements  of  external  stimuli  that  formed  the  instructional  environment.  In  the  1980s  and 
1990s,  more  energy  was  focused  on  identifying  the  processes  that  occur  internal  to  the  individual.  Although  many 
questions  remain  unresolved,  countless  educational  researchers  view  learners  as  active  participants  in  their  learning, 
and  believe  that  the  role  of  a teacher  should  be  to  facilitate  the  learner’s  efforts  to  make  meaning  of  their  world 
(Jonassen,  Beissner,  & Yacci,  1993;  Jonassen  &Tessmer,  1996;  Wittrock,  1974,  1990,  1991,  1992). 

But  how  does  a teacher  facilitate  knowledge  acquisition?  According  to  Osborne  and  Wittrock  (1985), 
teachers  need  to  provide  opportunities  for  students  to  construct  their  own  knowledge  and  to  reflect  on  their  freshly 
generated  views.  A common  teaching  strategy  is  to  create  an  environment  in  which  students  analyze  new  material, 
provide  opportunities  for  students  to  combine  their  new  ideas  with  previously  constructed  knowledge,  and  facilitate 
student  articulation  on  how  the  new  information  fits  in  with  their  existing  mental  structures.  As  this  occurs,  new 
mental  structures  are  created,  or  existing  structures  clarified.  The  teacher’s  role  then  is  to  assist  the  students  by 
providing  additional  information  to  either  clarify  the  knowledge  structure  if  misconceptions  are  obvious  or  to  enrich, 
enlarge,  and  strengthen  their  newly  developed  mental  structures. 

Although  students  are  continually  going  through  the  process  of  constructing  and  molding  their  knowledge  networks, 
teachers  often  structure  events  to  facilitate  this  process.  Many  of  these  activities  have  their  roots  in  a learning  theory 
originally  articulated  by  Merlin  Wittrock,  known  as  generative  learning  theory  (Grabowski,  1996;  Wittrock,  1974). 

Types  of  Generative  Learning  Strategies 

The  fundamental  concept  upon  which  generative  learning  strategies  is  based  is  that  learning  involves  the 
creation  and  refinement  of  individual  mental  constructions  of  the  world.  Thus,  facilitating  that  construction  should  be 
a primary  concern  for  teachers.  On  the  other  hand,  not  all  class  work  is  considered  generative  in  nature,  even  though 
it  might  help  students  build  and  clarify  their  conceptualization  of  the  environment.  Grabowski  (1996)  informs  us  that 
only  those  activities  that  involve  the  actual  creation  of  relationships  and  meaning  are  classified  as  generative  learning 
strategies,  and  that  there  are  two  basic  families  of  these  strategies. 

One  type  of  strategy  is  used  to  generate  organizational  relationships  between  different  components  of  the 
environment,  which  helps  a learner  understand  how  items  are  related  to  one  another.  Well  documented  examples 
include  creating  titles,  headings,  questions,  objectives,  summaries,  graphs,  tables,  and  concept  maps.  On  the 
periphery  of  this  group  is  the  manipulation  of  objects  which  some  argue  may  also  qualify  as  a generative  strategy, 
“because  a relationship  is  being  drawn  and  extended  between  parts  of  the  environment”  (Grabowski,  1996,  p.  91 1). 

A second  type  of  generative  activity  integrates  relationships  between  external  stimuli  and  memory.  Examples  include 
asking  students  to  construct  demonstrations,  metaphors,  analogies,  examples,  pictures,  applications,  paraphrases,  or 
inferences.  This  group  differs  from  the  previous  group  in  that  these  strategies  not  only  require  deeper  processing  of 
the  instructional  content,  but  they  also  result  in  a higher  level  of  understanding. 
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Problem  statement 


Although  generative  strategies  have  been  found  to  improve  the  effectiveness  of  learning  (for  a discussion  of 
the  studies  on  generative  learning  see  Wittrock,  1992,  or  Grabowski,  1996),  it  cannot  be  assumed  that  all  generative 
strategies  work  equally  well  under  all  conditions,  and  questions  regarding  the  selection  of  appropriate  strategies  or 
how  best  to  implement  them  are  not  yet  sufficiently  solved. 

One  question  related  to  the  instructional  setting  is  whether  students  learn  better  as  individuals  or  in  groups. 
Kourilsky  and  Wittrock  (1992)  found  that  cooperative  learning  environments  foster  generative  learning.  Based  on 
their  study  with  12th  grade  economics  students,  they  reported  that,  “Cooperative  learning  appeared  to  provide  an 
excellent  context  for  students’  discovery  of  one  another’s  misconceptions  and  for  the  generation  of  alternative 
relations  that  better  synthesize  experience  within  the  concepts  of  economics”  (p.  874).  However,  the  success  of 
cooperative  learning  depends  on  effective  small  group  interaction,  which  cannot  be  assumed  for  all  groups.  Hooper, 
Sales,  and  Rysavy  (1994)  replicated  a study  conducted  by  Wittrock  and  Alesandrini  (1990)  examining  the  effects  of 
generative  learning  strategies  on  text  comprehension.  Hooper  et  al.  attempted  to  extend  the  original  study  by 
investigating  the  effects  of  using  a generative  learning  technique  while  studying  alone  or  in  small  groups. 
Unexpectedly,  they  found  that  students  working  alone  achieved  better  results  on  a posttest  than  students  learning  in 
groups.  The  researchers  attributed  this  effect  mostly  to  deficiencies  in  group  interactions. 

The  instructional  setting  may  influence  the  selection  and  effectiveness  of  appropriate  generative  learning 
strategies  in  other  ways  as  well.  For  example,  science  and  math  teachers  often  face  different  requirements  and 
challenges  in  terms  of  resources,  classroom  organization,  etc.  when  conducting  experiments  that  require  specific 
apparatus  as  compared  to  traditional  teaching  activities.  Under  such  conditions,  the  additional  efforts  can  only  be 
justified  through  improvements  in  the  expected  outcomes.  As  stated  earlier,  manipulating  these  sorts  of  objects  may 
also  qualify  as  a generative  strategy  and  therefore  justify  the  additional  efforts,  yet  little  research  has  been  done  in 
this  area. 

Sayeki,  Ueno,  and  Nagasaka  (1991)  investigated  the  effectiveness  of  using  a manipulative  strategy  to  teach 
Japanese  elementary  students  how  to  calculate  the  area  of  a parallelogram.  The  way  students  used  the  mediational 
objects  appears  to  qualify  as  a generative  learning  strategy.  Sayeki  et  al.  reported  that  the  use  of  the  manipulatives 
increased  students’  comprehension  of  mathematical  principles.  Might  it  be  possible,  then,  that  manipulative  activities 
can  be  truly  considered  generative,  and  successful  in  other  content  areas  and  for  other  types  of  learning  outcomes? 
This  question,  along  with  the  question  of  whether  an  individual  or  team  approach  works  better  for  generative 
learning  environments,  became  the  driving  forces  of  our  study. 

Purpose  of  Study  and  Hypotheses 

The  purpose  of  this  study  was  to  assess  the  effectiveness  of  two  generative  learning  strategies,  concept  maps 
and  manipulation  of  objects,  with  sixth-grade  students,  and  to  determine  if  either  strategy  works  better  with 
individual  learners  or  groups  of  learners. 

Two  research  hypotheses  were  generated: 


• Based  on  the  notion  that  the  manipulative  activity  of  completing  a science  experiment  acts  as  a generative  learning 
strategy,  students  who  use  this  method  will  obtain  criterion  achievement  scores  equal  to  those  of  students  developing 
concept  maps  on  the  same  topic. 

• Students  who  work  in  teams  will  score  significantly  higher  on  criterion  achievement  tests  than  students  who  work 
individually. 


Methodology 
Population  and  sample 

The  accessible  population  was  sixth-grade  science  students  enrolled  in  a middle  school  in  San  Diego, 
California.  The  cluster  sample  of  80  students  included  all  sixth  grade  students  at  this  school.  Three  students  dropped 
out  of  the  experiment  between  the  second  test  and  the  delayed-post  test. 

Materials 

A chapter  from  the  Prentice  Hall  Science  textbook  Exploring  Earth  Science  (Maton,  Hopkins,  Johnson, 
LaHart,  Warner,  & Wright,  1995)  served  as  basis  of  the  subject-matter  content.  Knowledge  acquisition  was 
measured  by  a 13-question  multiple-choice  test.  Questions  for  the  test  came  from  the  chapter  test  and  additional 
questions  were  generated  to  match  the  content  area.  The  classroom  teacher  worked  with  the  researchers  during  the 
construction  of  the  additional  test  questions,  and  verified  the  questions’  face- validity. 
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Treatment  and  Procedure 

Students  were  randomly  assigned  to  one  of  four*  treatment  groups  using  the  class  list  and  a table  of  random 
numbers.  The  groups  included  students  working  with  concept  maps  in  a team  situation  (CT),  students  working  with 

334 


315 


concept  maps  individually  (Cl),  students  working  with  manipulates  in  a team  situation  (MT),  and  students  working 
with  manipulates  individually  (MI).  Each  of  the  four  classrooms  had  approximately  equal  numbers  of  students  in 
each  of  the  four  groups. 

Each  classroom  participated  in  parallel  instruction  provided  by  the  same  teacher  and  textbook.  At  the 
conclusion  of  the  first  chapter  segment  (about  1/3  of  the  way  through  the  chapter),  students  in  the  CT  and  Cl  groups 
were  separated  and  moved  to  another  room.  There,  they  were  given  a blank  sheet  of  paper.  They  were  then  asked  to 
generate  a concept  map  that  describes  nine  characteristics  of  minerals  (the  topic  they  had  just  studied).  All  students 
were  familiar  with  making  concept  maps  as  they  had  made  numerous  concept  maps  earlier  in  the  school  year. 
Students  assigned  to  the  CT  group  created  the  concept  maps  in  teams  of  two  or  three  students,  while  the  students 
assigned  to  Cl  created  concept  maps  individually. 

Students  assigned  to  groups  MT  and  MI  were  separated  and  given  a set  of  hardness  points,  a Moh’s 
hardness  scale  table,  a mineral  color  chart,  and  a set  of  six  minerals.  The  students  were  asked  to  use  the  equipment 
and  charts  to  identify  each  mineral  sample.  Students  in  MT  performed  the  experiment  in  teams  of  two  or  three 
students,  and  students  in  MI  conducted  the  experiment  working  individually.  Students  in  all  four  groups  (CT,  Cl, 
MT,  and  MI)  were  allowed  to  use  the  textbook  as  support  material. 

The  following  day  all  students  were  assessed  on  their  knowledge  of  the  content  with  use  of  the  instrument 
described  above. 

At  the  conclusion  of  the  chapter  (10  days  later),  students  who  previously  created  concept  maps  were 
assigned  to  conduct  the  manipulatives  experiment,  and  students  who  conducted  the  experiment  created  concept  maps 
(the  CT  group  switched  activities  with  MT,  and  the  Cl  group  switched  activities  with  MI).  To  allow  testing  of  the 
second  hypothesis,  students  who  worked  in  teams  during  the  initial  activity  continued  to  work  in  teams  during  the 
second  activity  (although  they  switched  between  concept  mapping  and  manipulatives).  The  next  day  a chapter  test 
was  conducted,  including  the  same  questions  as  used  during  the  initial  quiz.  Thirty-two  days  later  all  students 
completed  a delayed  posttest  which  was  part  of  a larger  unit  test. 

Results 

Descriptive  Statistics 

The  group  names,  number  of  individuals,  means,  and  standard  deviations  of  the  results  for  each  of  the  three 
tests  are  shown  in  Table  1.  Note  that  the  differences  in  the  number  of  individuals  (n)  are  caused  by  the  variance  in 
class  size  combined  with  variations  due  to  using  the  table  of  random  numbers. 

Table  1.  Descriptive  statistics  for  the  three  tests 


Group  names  based  on  first  test 

Test  1 

(Following  day) 

Test  2 

(10  days  later) 

Test  3 
(32  days  later) 

n 

M 

SD 

n 

M 

SD 

n 

M 

SD 

Manipulatives,  Teams  (MT) 

18 

10.72 

2.35 

18 

11.11 

1.97 

18 

10.56 

2.23 

Concept  Maps,  Teams  (CT) 

22 

11.95 

1.25 

22 

12.05 

1.00 

21 

12.14 

1.11 

Manipulatives,  Individual  (MI) 

19 

11.32 

1.53 

19 

12.05 

1.39 

19 

11.63 

1.80 

Concept  Maps,  Individual  (Cl) 

21 

10.9 

1.58 

21 

11.05 

1.47 

19 

11.47 

1.78 

Concept  Maps,  Total 

43 

11.44 

1.50 

43 

11.56 

1.74 

40 

11.82 

1.48 

Manipulatives,  Total 

37 

11.03 

1.97 

37 

11.56 

1.33 

37 

11.11 

2.07 

Teams,  Total 

40 

11.40 

1.90 

40 

11.63 

1.56 

39 

11.41 

1.87 

Individuals,  Total 

40 

11.10 

1.55 

40 

11.53 

1.50 

38 

11.55 

1.77 

Inferential  Statistics 

The  data  were  statistically  analyzed  using  a 2 X 2 factorial  analysis  of  variance  (ANOVA),  with  a level  of 
significance  set  at  .10.  The  level  of  significance  was  set  at  .10  because  little  research  has  been  done  comparing 
generative  learning  strategies  (therefore  this  could  be  considered  an  exploratory  study)  and  we  felt  it  better  to  make  a 
Type  1 error  than  a Type  2 error.  The  two  manipulated  variables  under  study  included  the  types  of  generative 
learning  strategy  (concept  maps  and  manipulatives)  and  the  method  of  instruction  (individual  and  team).  Besides 
comparing  the  initial  test  results,  a secondary  analysis  was  conducted  to  identify  if  any  changes  in  long-term  memory 
occurred  that  may  have  been  dependent  on  the  treatments. 

The  following  charts  report  the  F and  p values  from  the  three  tests: 
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Table  2.  ANOVA  table  for  Test  1 


Variable 

DF 

Sum  of  Squares 

Mean  Square 

F-Value 

P-Value 

Concept  Map  / Manipulatives 

1 

3.352 

3.352 

1.166 

.2837 

Team  / Individual 

1 

1.034 

1.034 

.360 

.5504 

Concept  Map  / Manipulatives  * Team/ 
Individual 

1 

13.418 

13.418 

4.668 

.0339 

Residual 

76 

218.480 

2.875 

Table  3.  AN  OVA  table  for  Test  2 


Variable 

DF 

Sum  of  Squares 

Mean  Square 

F-Value 

P-Value 

Concept  Map  / Manipulatives 

1 

.25 

.25 

Oil 

.9150 

Team  / Individual 

1 

.016 

.016 

.007 

.9323 

Concept  Map  / Manipulatives  * Team/ 
Individual 

1 

18.688 

18.688 

8.627 

.0044 

Residual 

76 

164.632. 

2.166 

Table  4.  ANOVA  table  for  Test  3 


Variable 

DF 

Sum  of  Squares 

Mean  Square 

F-Value 

P-Value 

Concept  Map  / Manipulatives 

1 

9.802 

9.802 

3.192 

.0781 

Team  / Individuals 

1 

.794 

.794 

.259 

.6126 

Concept  Map  / Manipulatives  * Team/ 
Individual 

1 

14.612 

14.612 

4.758 

.0324 

Residual 

76 

224.174 

3.071 

Concept  Maps  and  Manipulatives 

For  both  Test  1 and  Test  2,  no  significant  difference  was  found  between  either  of  the  two  generative 
learning  strategies  (manipulatives  and  concept  maps).  This  supports  the  first  hypothesis  that  students  engaged  in  a 
manipulative  activity  will  achieve  scores  equal  to  those  of  students  developing  concept  maps  on  a similar  topic. 

The  delayed  posttest,  however,  revealed  a statistically  significant  difference  between  concept  maps  and 
manipulatives,  F(l,76)  = 3.192,  p<.10.  This  result  identifies  a statistically  significant  difference  in  long-term 
retention  scores  among  students  who  began  with  concept  maps  ( M = 1 1.82)  compared  to  students  who  initially 
worked  with  manipulatives  (M  = 11.11). 

Individual  Learning  and  Collaborative  Learning 

No  significant  differences  were  found  among  students  who  worked  individually  versus  collaboratively  in 
any  of  the  three  tests.  This  finding  fails  to  support  the  second  hypothesis  that  students  who  work  in  teams  would 
score  significantly  higher  than  students  who  work  individually. 

Interaction  between  Strategies  and  Organization 

For  each  of  the  three  tests,  the  factorial  ANOVA  revealed  a statistically  significant  interaction  between 
concept  maps/manipulatives  and  teams/individual.  In  Tests  1,  2,  and  3,  the  respective  F values  were:  F(l,76)  = 
4.668,  p<.  10;  F(l,76)  = 8.627,  p<.  10;  F(l,76)  = 4.758,  p<.10.  The  graphs  below  display  the  interactions  for  each 
test. 
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Figure  I:  Interaction  Line  Plot  for  Test  1 


Figure  2:  Interaction  Line  Plot  for  Test  2 


Figure  3:  Interaction  Line  Plot  for  Test  3 


T 

I 


Discussion 

As  mentioned  earlier,  the  study  verified  our  first  hypothesis  in  the  first  and  second  test — that  students 
engaged  in  a manipulative  activity  will  achieve  scores  equal  to  those  of  students  developing  concept  maps  on  a 
similar  topic.  This  lends  support  that,  at  least  in  this  case,  the  two  strategies  work  equally  well.  When  the  equality  of 
strategies  is  added  to  the  idea  that  the  manipulation  of  mediational  objects  is  helping  learners  to  create  mental 
relationships  and  meaning  between  knowledge  components,  it  supports  the  idea  that  manipulation  of  items  may  be 
considered  a generative  learning  strategy. 
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The  study  failed  to  confirm  the  second  hypothesis  (i.e.,  students  learn  better  in  teams  than  as  individuals). 
Interestingly,  when  these  two  variables  were  analyzed  together,  we  found  statistically  significant  interactions 
between  the  generative  learning  strategies  (concept  maps  and  manipulative  experiments)  and  the  classroom 
organization  (individuals  and  teams)  for  each  of  the  three  tests. 

These  treatment  interactions  cannot  be  easily  explained  because  they  seem  to  be  in  reverse  order  between 
Test  1 and  Test  2.  For  example,  on  the  first  test  teams  creating  concept  maps  (CT)  showed  higher  mean  scores  (M  = 

1 1 .95)  than  their  counterparts  in  the  manipulative  groups  (MT)  (M  = 10.72).  But  in  the  second  test,  teams  conducting 
the  manipulative  experiments  (MT)  achieved  higher  mean  scores  ( M = 12.05)  than  teams  creating  concept  maps 
(CT)  (M  = 11.11).  Because  the  same  group  of  students  outscored  the  other  group  no  matter  the  generative  strategy 
imposed,  this  led  us  to  originally  conclude  that  the  results  might  be  better  explained  by  parameters  such  as  pre-test 
performance,  general  aptitude,  or  student  learning  style  than  by  the  experimental  treatments.  However,  the  delayed 
posttest  yielded  a treatment  interaction  again,  but  this  time  in  combination  with  a significant  difference  between 
mean  scores  achieved  by  students  working  on  concept  maps  (Af  = 1 1.82)  versus  students  engaged  in  the  manipulative 
experiments  (Af  = 11.11). 

Before  we  present  what  we  consider  a potentially  noteworthy  finding,  we  feel  it  important  to  acknowledge 
the  limitations  of  the  study.  Our  finding  could  simply  be  a consequence  of  the  experimental  design,  which  may  not 
have  addressed  all  threats  to  external  validity  (i.e.  multiple  treatment  interaction).  Another  limitation  was  that 
although  all  students  were  in  the  class  for  the  instruction,  there  was  a difference  in  information  imparted  between  the 
test  sessions,  and  this  difference  may  have  influenced  student  scores.  Additionally,  it  can  be  said  that  students  may 
not  have  received  enough  specific  instruction  regarding  collaborative  learning  techniques.  This  reminds  us  of  the 
study  of  Hooper,  Sales,  and  Rysavy  (1994)  who  pointed  out  that  one  cannot  simply  assume  that  effective  small  group 
interaction  occurs  naturally  without  teacher's  guidance. 

A more  tantalizing  explanation  is  that  the  sequence  of  generative  learning  strategies  may  be  responsible  for 
the  differences  we  found.  Students  who  started  with  concept  maps  and  then  switched  to  manipulative  experiments 
retained  information  better  than  those  who  started  with  experiments  and  then  did  concept  maps.  Interestingly,  the 
mean  scores  of  students  who  started  with  concept  maps  were  higher  on  the  delayed  post-test  than  on  their  first  test, 
whereas  the  mean  scores  of  students  who  started  with  manipulative  experiments  were  lower  on  the  delayed  post-test 
than  on  the  initial  test. 

These  results  appear  to  indicate  that  creating  concept  maps  prior  to  engaging  in  manipulative  experiments 
produces  better  long-term  retention  than  the  opposite  sequence.  Or  to  over  generalize  for  a moment,  better  results  are 
found  when  teachers  use  more  inclusive,  encompassing,  or  general  activities  before  specific  or  concrete  activities. 
This  finding  supports  Reigeluth's  elaboration  theory  (English  & Reigeluth  1996;  Reigeluth,  1983),  which  claims  that 
it  is  important  to  explicitly  state  the  overall  structure  of  the  content  before  getting  into  the  details.  Elaboration  theory 
proposes  a simple-to-complex  sequence  “with  the  most  essential  information  presented  in  advance  of  more  complex 
and  detailed  information”  (Jonassen,  Beissner,  & Yacci,  1993,  p.  1 16). 

Why  does  the  sequence  make  a difference?  Translated  to  the  framework  of  this  study,  one  could  argue  that 
the  students'  mental  models  achieved  during  the  concept  map  activity  were  more  expansive  than  mental  models 
achieved  while  the  students  were  focusing  on  the  details  during  the  manipulative  experiment.  Thus,  learners  creating 
concept  maps  first  may  have  established  a better  network  to  which  they  attached  the  small  chunks  of  information 
encountered  later  during  the  manipulative  experiment. 

Contrarily,  doing  the  concrete  activity  first  (i.e.,  the  manipulative  experiment),  some  of  the  facts  may  have 
seemed  disassociated  with  the  other  pieces;  or — to  say  it  metaphorically — students  couldn’t  see  the  forest  for  the 
trees.  They  learned  little  things,  but  didn't  get  the  big  picture.  Grasping  the  big  picture  first  through  the  use  of 
concept  maps  provided  the  expansive  view,  to  which  they  attached  the  individual  facts  when  they  did  the 
experimental  activity. 

In  conducting  the  literature  review  for  this  study,  we  were  only  able  to  identify  one  previous  study  that 
examined  the  sequence  of  generative  learning  strategies  in  regards  to  knowledge  acquisition.  In  a study  investigating 
the  effects  of  generative  strategies  on  reading  comprehension.  Linden  and  Wittrock  (1981)  compared  sequences  of 
learner  activities  that  proceeded  from  imaginal  to  verbal  generations  with  sequences  of  activities  in  the  reverse  order. 
They  concluded  in  their  study  that  the  sequence  wasn’t  critical,  and  that  “. . .it  seems  that  the  generation  of  text- 
relevant associations,  in  either  order,  was  the  more  important  factor  in  enhancing  comprehension  in  this  study" 
(Linden  and  Wittrock,  1981,  p.  55).  However,  with  the  results  of  this  study  indicating  that  sequence  is  important,  and 
recent  explorations  into  learning  proposed  by  the  elaboration  theory,  further  inquiry  into  this  topic  is  warranted. 
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Abstract 

Problem-based  learning  (PBL),  initiated  when  learners  meet  the  problem  scenarios , develops  learners' 
skills  and  subject-matter  contents  to  make  the  transition  from  beginner  to  more  expert  problem  solver. 
Beginning  students  might  benefit  from  simple,  well-structured  problems,  in  which  the  understanding  of  sets 
of  rules  and  principles  result  in  a straightforward  solution.  And  for  learners  with  more  experience, 
facilitators  may  consider  to  provoke  them  with  complex,  ill-structured  problems.  However,  seldom  have 
reports  provided  PBL  facilitators  the  solutions  for  how  to  design  or  select  problem  scenarios  so  that  it 
opens  the  door  to  significant  learning  outcomes  consistent  with  course  or  program  goals.  The  gray  area 
between  well-structured  problems  and  ill-structured  problems  need  to  be  further  clarified  for  better 
learning  effect.  After  a brief  literature  review,  the  author  outlines  a decision  model  for  problem  selection. 
Conclusion  suggests  that  a PBL  facilitator  should  consider  the  learning  strategies  both  systematically  and 
systemically.  Future  research  is  required  to  further  develop  this  decision  model. 

Introduction 

From  the  instructor’s  viewpoint,  the  most  important  part  of  problem  based  learning  might  be  the  problem 
selection.  Selecting  and  crafting  the  problems  represents  the  major  way  in  which  the  instructor  plans  the  learning 
and  also  portrays  the  course  goals  that  the  instructor  wants  to  reach.  No  matter  whether  instructors  adapt  a well- 
structured  problem  or  an  ill-structured  problem,  once  the  students  confront  a problem,  they  will  start  reacting  and 
take  ownership  for  their  own  learning.  Thus,  Instructors  must  carefully  conceive  the  problem  scenarios  before  giving 
them  to  students  (Stepien  W.  & Fyke  S.  L.,  1997). 

This  study  will  start  describing  some  conceptual  issues  regarding  the  nature  of  problems  and  their  roles  in 
the  environment  of  problem  based  learning.  A brief  analysis  of  whether  a we  11 -structured  problem  or  an  ill -structured 
problem  could  better  fulfill  students’  needs  will  be  presented.  Lastly,  the  detailed  explanation  of  the  decision  model 
of  problem  selection  will  be  categorized  in  these  three  dimensions:  Conditions,  Methods,  and  Outcomes. 

The  problem  of  problem  based  learning:  A few  conceptual  issues 
Nature  of  the  problem 

In  most  PBL  environments,  two  different  structures  of  problem  scenarios  are  used:  well-structured  problem 
scenarios  and  ill-structured  problem  scenarios.  They  differ  from  one  another  in  significant  ways. 

1 . Well -structured  problem  scenarios 

A well -structured  problem  scenario  is  tidy  and  has  little  complexity.  It  is  well  organized  and  may  lead  students  to  a 
straightforward  solution.  Naives  might  benefit  from  simple,  well-structured  problems,  in  which  the  understanding 
of  a set  of  rules  and  principles  leads  to  a straightforward  solution.  Bridge  (1995)  describes  the  well-structured 
problem  as  “The  type  of  problem  that  most  instructors  encounter  annually  and  has  certain  steps  for  solving  the 
problem”. 

2,  111 -structured  problem  scenarios 

An  ill -structured  problem  scenario  is  formed  by  undefined  problems  and  incomplete  information.  It  simulates  a 
real-world  situation.  Because  it  has  no  single  path  to  solution  (Mason  & Mitroff,  1988),  students  will  need  to 
explore  the  situation  they  have  been  given  by  building  hypotheses  that  initiate  inquiry  into  the  numerous  aspects  of 
the  problem.  Bridge  (1995)  defines  an  ill -structured  problem  that  is  complex,  messy,  and  without  clear  goals.  With 
the  belief  of  authenticity  and  fidelity  (Brown  & Collins,  1989;  Dyson,  1993;  Duffy  & Savery,  1994)  to  resemble 
situations  in  the  real  world,  when  students  confront  a ill-structured  problem,  they  will  not  have  most  of  the  relevant 
information  needed  to  solve  the  problem  at  the  outset.  Nor  will  they  know  exactly  what  actions  are  required  for 
resolution.  And  even  after  they  propose  a solution,  the  students  will  never  be  sure  that  they  have  made  the  right 
decision. 

Simon  (1978)  indicated  three  characteristics  to  explain  the  differences  between  well-structured  and  ill- 
structured  problems.  First  of  all,  in  contrast  to  an  ill-structured  problem,  a well-structured  problem  could  be  solved 
by  correct  methods  which  are  obvious  and  do  not  conflict.  An  ill-structured  problem  often  means  a complex  inquiry 
process  before  sets  of  hypotheses  and  solutions  can  be  formed.  The  secondary  characteristic  is  the  availability  of 
information  for  diagnosis.  A well-structured  problem  is  well  organized,  containing  the  information  needed  for 
solutions.  Namely,  students  are  led  to  instructors’  desired  outcomes.  On  the  other  hand,  an  ill-structured  problem  is 
mess  and  disorder.  It  supplies  less  than  enough  information  and  requires  careful  inquiry  and  numbered  trials  before 
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students  can  finalize  the  appropriate  solutions.  Lastly,  the  third  characteristic  Simon  proposed  to  differentiate  well- 
and  ill-structured  problems  is  “legal  move  generators”  (Simon,  1978).  In  a we  11 -structured  problem  scenario,  routes 
and  procedures  are  pre-defined  by  instructors.  Unlike  an  ill-structured  problem,  the  solutions  for  the  problem  are  not 
constrained.  An  ill-structured  problem  simulates  an  authentic  situation  as  in  the  real  world,  and  the  “problem  space” 
(Voss,  J.  F.,  1989)  is  wide  enough  for  people  with  various  backgrounds  come  up  with  different  solutions  for  the 
same  problem. 

However,  the  way  Simon  distinguished  well-structured  and  ill-structured  problems  is  to  place  each  of  them 
at  the  opposite  end  of  a continuum  (Simon,  1978).  The  ambiguous  area  for  the  spectrum  of  problem  representation, 
is  still  an  open  question  for  instructors  and  researchers. 

Well-structured  VS  Ill-structured  problem 

Some  studies  advocate  that  in  problem  based  learning,  ill-structured  problems  could  serve  all  learners’ 
needs  no  matter  their  abilities  are  in  what  levels.  A naive  learner  still  could  benefit  from  solving  a problem  with  mess 
situation,  incomplete  information,  and  undefined  questions.  However,  a PBL  facilitator  still  needs  to  consider  the 
scope  of  a problem.  It  should  be  reasonable  enough  for  naive  learners  without  much  experience  to  complete  within 
limited  time  and  could  also  achieve  the  goals  fulfilled  their  needs  and  facilitators’  expectations. 

Well-structured  problems  might  be  still  worthwhile  for  certain  situations  during  the  PBL  process.  The 
author  would  like  to  analyze  this  from  an  epistemological  viewpoint.  PBL  is  a learning  approach  based  on  the  theory 
of  constructivism.  With  the  belief  of  constructivism,  learning  is  a construction  process.  Learners  have  their  unique 
experience  to  the  world.  Some  constructivists  may  argue  that  there  could  be  no  ultimate,  shared  reality  (Duffy  & 
Jonassen,  1992)  and  only  has  best  “description”.  And  the  best  “description”  is  developed  through  social  negotiation 
(Vygotsky,  1978).  Instructors  should  not  have  the  objectives  predetermined  and  contents  bounded.  However,  this 
issue  might  not  apply  to  each  level  of  learning.  However,  in  Dick’s  study  (1991),  he  argues  that  constructivism  is 
compatible  with  system  theory.  Constructivism  also  could  be  a micro  level  theory  in  a systematic  process.  Schwen  et 
al.  (1993)  advocates  a similar  idea.  In  the  article  Enriched  Learning  and  Information  environment , they  proposed  the 
small  “c”  constructivism  by  arguing  that  instructors  can  put  the  “idiosyncratic”  intellectual  thinking  of  experts  into 
the  instruction  and  allow  learners  to  select  according  to  their  needs.  It  has  been  suggested  that  learners  at  different 
learning  statuses  might  require  different  instructional  strategies. 

Jonnason  (1991)  points  out  that  there  are  three  levels  of  learning  stages:  Introductory,  Advanced,  and 
Expert.  In  the  Introductory  stage,  what  learners  need  is  the  instruction  closer  to  objectivist.  And  in  the  advanced  or 
expert  stages,  the  constructivist  approach  may  become  more  appropriate.  This  argument  was  supported  by  Ertmert’s 

Figure  4.  ( Ertmert , p69>  1993) 
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study.  Ertmert  et  al.  (1993)  indicate  that  there  is  no  single  theory  that  can  fulfill  the  needs  of  entire 
instruction  design  (Figure  4).  As  learners  acquire  more  knowledge  and  skills,  they  will  progress  along  the  knowledge 
continuum  from  low  to  high.  That  is,  learners  will  move  up  from  the  knowledge  about  “knowing  what”,  to  “knowing 
how”, 

and  eventually  achieve  the  “reflection-in-action”  level.  Learning  strategies  should  be  chosen  according  to 
where  the  learners  “sit”  on  the  continuum  in  terms  of  their  knowledge  and  skills. 
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Therefore,  PBL  could  still  be  compatible  with  the  systematic  learning  process.  In  other  words,  a teacher 
could  still  be  an  instructor  role  for  the  naive  user  during  the  process  of  PBL.  If  it  is  the  need  of  students,  a lecture  of 
a well-structured  problem  could  be  helpful  for  them  to  establish  the  basic  knowledge  domain.  Beginning  learners 
could  benefit  from  a simple,  we  11 -structured  representation  (Wilkerson,  L & Geletti,  G.).  And  learning  objectives  for 
what  needs  to  be  learned  should  be  decided  jointly  by  teachers,  students,  and  even  parents.  In  this  way,  the 
transmission  and  adoption  become  desirable.  Students  will  no  longer  feel  what  all  they  do  are  for  teachers’ 
requirements  only.  In  general,  teachers  should  always  remember  that  to  provide  a well-structured  problem  is  for 
students’  desires  not  for  fulfilling  teachers’  expectation. 

The  decision  model  of  problem  selection 

Based  on  above  assumptions,  this  decision  model  of  problem  selection  (Figure  2)  is  organized  and 
explained  by  the  following  three  dimensions:  Conditions,  Methods,  and  Outcomes  (Reigeluth,  1983).  Conditions  are 
defined  as  the  factors  that  can  not  be  controlled  by  instructors,  while  they  still  have  a cause-effect  relation  with 
Methods.  Methods  are  manageable  and  represent  various  ways  to  achieve  desired  outcomes  under  different 
conditions.  Finally,  outcomes  could  be  viewed  as  “various  effects  that  provide  a measure  of  the  value  of  alternative 
methods  under  different  conditions”  (Reigeluth,  pi 5,  1983).  In  terms  of  these  three  dimensions  in  this  study,  the 
Conditions  stage  is  to  provide  the  facilitator  with  the  things  that  need  to  be  considered  during  the  diagnosis  process. 
The  Methods  stage  is  to  provide  facilitators  the  available  strategies  and  techniques  which  could  reflect  the  needs  of 
Conditions.  The  Outcomes  stage  is  the  criteria  of  well-structured  problems  and  ill- structured  problems  listed  from 
simple  to  complex.  PBL  facilitators  will  be  able  to  choose  the  appropriate  problems  by  thinking  through  the  stages  of 
Conditions  and  Methods. 


Figure  2.  The  decision  model  of  problem  selection 
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As  Stepien  et  al  (1997)  indicate,  to  identify  an  appropriate  problem,  a PBL  facilitator  should  consider 
whether  it  (a)  contains  significant  content;  (b)  fits  curriculum  outcomes  for  a specific  course  or  program;  (c)  is 
appropriate  for  the  targeted  audience;  and  (d)  can  be  managed  effectively  by  students  in  the  time  available  to  them. 

Bridges  & Hallinger  (1995)  also  propose  a similar  idea  that  a teacher  should  consider,  such  as  (a)  learning 
objectives;  (b)  prerequisite  skills  and  knowledge;  (c)  available  time  and  time  constraint;  and  (d)  presentation  formats 
when  trying  to  choose  a problem.  Accordingly,  the  author  adopts  these  four  factors  and  puts  them  in  the  Conditions 
step.  It  is  suggested  that  PBL  facilitators  base  their  diagnosis  in  correspondence  with  these  factors  in  choosing  an 
appropriate  problem. 

Learning  Objectives  (course  goals) 

For  facilitators  of  PBL,  their  consideration  is  whether  the  learning  objectives  are  appropriate  to  the  course 
goals.  Certainly,  for  some  cases  of  PBL,  the  fit  between  learning  objectives  and  course  goals  may  not  be  quite  as 
straightforward  as  it  sounds.  Thus,  facilitators  and  learners  should  jointly  discuss  and  finalize  the  learning  objectives. 
It  needs  to  be  highlighted  here  that  in  a PBL  context,  setting  the  learning  objectives  is  a continually  rolling  process 
during  the  class  period.  It  is  not  a must  to  have  learning  objectives  concretize  before  the  class  while  PBL  facilitators 
could  seed  these  objectives  in  mind.  The  learning  objectives  that  need  to  be  designed  or  assembled  for  a problem 
solving  project  should  allow  “to  meet  the  individual  learning  needs  of  any  student  or  student  group  at  any  level, 
depending  upon  their  personal  career  goals,  ability,  and  background  knowledge”  (David  Bond,  1991). 


34 


BEST  COPY  AVAILABLE 


Prerequisite  skills  and  knowledge 

Since  the  learner’s  need  is  the  first  consideration  for  selecting  learning  objectives,  it  will  also  be  helpful  for 
both  instructors  and  learners  to  preview  whether  they  lack  any  of  the  prerequisite  skills  or  knowledge  that  are 
explicitly  needed  for  solving  problems.  If  results  do  show  the  needs,  instructors  should  act  as  learning  coaches  to 
guide  and  support  students  in  completing  learning  issues. 

Available  time  and  time  constraint 

The  instructor  must  consider  time  constraints  relevant  to  curriculum  implementation.  Time  constraints  will 
influence  how  complex  or  messy  a problem  should  be.  Instructors  can  expect  to  underestimate  the  time  students  need 
to  complete  a project.  Students  could  experience  how  they  manage  their  work  under  stress  and  time  pressure  the 
same  as  a situation  in  the  real  world.  In  fact,  it  is  suggested  that  learners  work  more  productively  in  more  substantial 
blocks  of  time  (Bridge,  1995).  Although  PBL  can  be  applied  in  a wide  variety  of  time  formats,  PBL  facilitators 
should  consider  the  recommended  duration  of  their  course  schedule  for  each  project  as  well  as  consider  the  format  of 
the  course,  such  as  the  number  of  times  and  length  of  class  meeting  per  week? 

Presentation  formats 

According  to  Bridges  (1995),  instructors  can  present  problems  as  a written  case,  a live  role-play,  interactive 
videodisk,  a taped  episode,  or  even  an  interactive  computer  simulation.  Each  different  delivery  “vehicle”  has 
different  functions.  PBL  facilitators  should  choose  the  most  suitable  way  to  transmit  singularly  different  characters  of 
problems. 

Method 

Strategy  1:  Simplifying  Conditions  Method  (SCM) 

Simplifying  Conditions  Method  is  a kind  of  sequence  strategy  that  provides  guidelines  to  process  course 
contents.  Reigeluth  (in  press)  indicates  that  “an  SCM  sequence  begins  with  the  simplest  version  of  the  task  that  is 
still  fairly  representative  of  the  task  until  the  desired  level  of  complexity  is  reached.”  In  other  words,  when  an 
instructor  decides  to  adapt  SCM  to  be  the  scaffolding  tool,  the  instruction  will  start  by  giving  students  the  simplest 
task  first.  Although  this  task  may  be  very  simple  and  basic,  it  still  includes  every  necessary  sub-skill  (basic  style)  and 
fair  representation  of  the  whole  task.  As  students  acquire  more  skills  and  content  knowledge,  the  next  task  will 
require  more  complex  or  deeper  skills  to  solve  until  students  achieve  the  desired  level  of  complexity.  SCM  does  not 
just  include  the  topical  fashion.  In  fact,  it  is  topical  fashion  with  the  concept  of  spiral  fashion. 

Technique  1:  Task  Expertise  & Content  Expertise 

There  are  two  different  versions  derived  from  SCM  according  to  their  different  characters.  First,  Task 
Expertise  is  the  sequence  that  “relates  to  the  learner  becoming  an  expert  in  a specific  task”(Reigeluth,  1995).  For 
instance,  a task  might  ask  students  to  perform  as  a good  writer. 

Two  subsequences  for  Task  Expertise  are  distinguished  as  follows  in  order  to  fit  different  course  contents 
(Reigeluth,  in  press). 

1.  Procedural  SCM  Sequence 

For  tasks  that  need  “a  set  of  steps  to  decide  when  to  do  what”  (Reigeluth,  in  press),  the  Procedural  Sequence  is 
appropriate  for  skill  development  task. 

2.  Casual  SCM  Sequence 

For  tasks  that  need  to  apply  “a  set  of  principle  to  decide  when  to  do  what”  (Reigeluth,  in  press),  the  Casual 
Sequence  is  appropriate  for  developing  tasks  like  thinking  skills  or  management  skills. 

Secondary,  Domain  Expertise  applies  to  the  situation  which  “relates  to  the  learner  becoming  an  expert  in  a 
body  of  subject  matter  not  tied  to  any  specific  task”  (Reigeluth,  in  press).  It  focuses  on  learners’  ability  in  the 
knowledge  domains.  Two  subsequences  are  further  defined  for  use  in  different  course  contents  (Reigeluth  & 
Darwazeh,  1982;  Reigeluth,  in  press). 

1.  Conceptual  Elaboration  Sequence 

Are  for  the  courses  which  focus  on  “interrelated  sets  of  concepts”  (Reigeluth,  in  press),  for  instance,  a Biology 
course  that  introduces  kinds  of  animals  or  plants  would  be  well  advised  to  apply  this  sequence. 

2.  Theoretical  Elaboration  Sequence 

Are  for  the  courses  which  focus  on  “interrelated  sets  of  principles”(Reigeluth,  in  press).  For  instance,  a Biology 
course  that  emphasizes  principles  of  life  cycles  and  bodily  functions  would  do  well  to  apply  this  sequence. 

Both  subsequences  under  Domain  Expertise  are  similar  to  those  two  under  Task  Expertise.  They  all  lead  to 
a gradual  progress  of  content  sequence  from  simple  to  complex  and  happen  concurrently  once  the  course  includes 
such  kinds  of  contents  described  above. 
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Strategy  2:  Problem' stimulated  strategy  & Student-centered  strategy 

PBL  can  be  divided  into  two  different  forms.  One  is  the  Problem-stimulated  form  and  the  other  is  the 
Student-centered  form  (Bond,  D & Felett,  G.,  1991).  The  major  differences  between  the  two  center  on  who  defines 
learning  objectives,  resources,  and  questions.  In  the  Problem-stimulated  form,  the  instructor  is  supposed  to  be 
responsible  for  identifying  learning  objectives,  providing  available  resources,  and  guiding  questions.  However, 
instructors  should  be  notified  that  they  still  act  as  a learning  coach  role  whereas  they  just  supply  more  directed 
guidance  in  this  stage.  In  contrast,  in  the  Student-centered  form,  students  are  obligated  to  be  responsible  for  these 
three  components.  That  is,  when  students  confront  a specific  problem,  they  are  able  to  identify  their  own  learning 
objectives,  locate  the  relevant  resources,  generate  hypothesis,  and  evaluate  solutions.  Although  Student-centered 
form  requires  fewer  instructors’  time  and  efforts,  there  are  certain  weaknesses  that  challenge  instructors.  First,  when 
given  the  opportunity  to  choose  their  own  learning  objectives,  students  may  pick  ones  that  only  partially  overlap  with 
those  objectives  considered  important  to  the  instructor.  Second,  since  students  are  generally  less  knowledgeable  than 
the  instructor,  they  may  fail  to  locate  the  correct  resources  within  the  available  time.  Third,  students  may  cover  less 
of  the  content  deemed  desirable  by  the  instructor. 

Technique  2: 

1 . The  selection  of  tasks  and  content  domains 

Instructors  need  to  determine  which  tasks  or  contents  should  be  combined  into  the  problems.  Instructors  will  need 
to  keep  asking  themselves:  ”Is  this  task  the  right  one  that  can  reflect  students’  needs?”  There  are  several  methods  to 
help  instructors  make  such  decision  so.  The  author  adopts  the  following  criteria  suggested  by  Bond,  D.  and  Felett, 

G.  ( 1991)  and  Bridges,  E.  M.  (1995).  Instructors  can  select  tasks  in  their  area  which  meet  these  characteristics: 

• Tasks,  or  conditions  that  have  the  greatest  frequency  in  the  usual  practice  setting 

• The  tasks  are  suitable  for  integrating  knowledge  from  a variety  of  disciplines 

• The  tasks  are  ones  that  have  high  potential  impact;  that  is,  they  affect  large  numbers  of  people  for  an  extended 
period. 

• The  tasks  might  expose  students  to  “discovered”  and  tlpresented”  problems. 

• The  tasks  that  emphasize  or  underline  important  basic  concepts  in  a specific  area 

Then,  instructors  can  rate  each  task  with  a score  from  one  to  five  based  on  each  of  three  different  factors: 
frequency,  potential  impact,  and  the  effectiveness  of  intervention  (Bond,  D.  & Felett,  G.,  1991).  With  the  multiplied 
product  of  these  scores,  instructors  can  determines  the  position  of  each  condition  on  a list.  Accordingly,  instructors 
will  know  which  task  or  when  it  should  be  selected. 

2.  Locating  resources 

Students  can  search  the  relevant  information  from  the  following  resources:  articles,  films,  computer  information 
networks,  and  consultants.  Besides,  since  students  often  bring  specialized  knowledge  and  skills  to  a problem- 
solving project,  they  should  be  encouraged  to  inventory  the  resources  existing  within  their  knowledge  background 
and  to  exploit  these  resources.  For  instance,  the  student  knows  a convenient  information  resource  that  is  seldom 
utilized  or  recognized  by  others. 

3.  Generate  questions 

In  the  Problem-stimulated  form,  PBL  facilitators  can  use  the  following  principles  to  guide  learners  forming 
questions  (Bridges,  1995): 

• to  direct  students  to  key  concepts, 

• to  assist  students  in  thinking  through  the  problem,  and 

o to  stimulate  students  to  view  the  problem  from  alternative  perspectives. 

For  the  Student-centered  form,  learners  discuss  in  a small  group  format  and  decide  which  learning  issues  to 
pursue,  and  subsequently  which  resources  to  utilize  in  order  to  obtain  the  necessary  information.  According  to  Duffy 
and  Cunningham  (1997),  this  step  is  not  complete  until  each  learner  has  an  opportunity  to  reflect  verbally  on  his  or 
her  position  in  the  problem,  and  to  assume  responsibility  for  some  of  the  learning  issues  identified.  Hence,  ownership 
extends  from  buying  into  the  problem  to  buying  into  the  learning  issues  identified.  The  process  to  generate  questions 
(learning  goals)  basically  fall  into  three  categories  (Stepien,  Gallagher,  & Workman,  1993;  Mardziah  et  al.,  1995): 


• What  do  we  (students)  know 

Information  always  confined  learners’  thought  as  they  face  problems  for  the  first  time  in  the  real  world.  This 
category  refers  to  learners’  prior  knowledge  that  may  be  applied  to  help  evaluate  the  ideas  generated.  At  this  point, 
learners  need  to  challenge  any  ideas  or  knowledge  presented  for  accuracy  and  understanding. 
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• What  do  we  (students)  need  to  know 

This  category  refere  to  topics  or  issues  that  need  further  investigation  to  yield  information  that  would  help  leamere 
evaluate  the  ideas  generated.  As  information  is  collected,  gaps  and  conflicts  within  the  problem  are  noted. 

• What  should  we  (students)  do 

Learners  will  start  to  analyze  and  revise  the  list  of  questions  under  the  heading  “What  do  we  need  to  know”.  The 
reasoning  process  will  keep  cycling  until  learners  are  satisfied  with  the  exact  questions  that  need  answering. 


Outcome 

As  Simon  (1987)  proposes  to  place  two  types  of  problems  (well-  and  ill-structured)  at  the  opposite  end  of  a 
continuum  to  describe  and  differentiate  them,  PBL  facilitators  might  feel  difficulties  in  forming  problems  without 
defining  the  “gray  area”  in  between.  A clear  classification  for  problems  with  various  levels  of  complexities  is 
considered  necessary.  Both  Bashook  (1976)  and  McGuire  (1980)  describe  valuable  systems  to  solve  this  problem. 
Berner  (1984)  values  Getzels’  dimension  (1982)  to  be  a major  structural  way  for  problem  classification.  Getzels’ 
problem  typology  attempts  to  categorize  well-structured  and  ill-structured  problems  into  four  levels. 

Well-structured  Problems 


1.  Level  one 

This  type  of  problem  is  that  both  information  needed  for  problem  diagnosis  and  appropriate  solutions  are  known  to 
the  problem  solver.  In  addition,  the  instructor  often  encounters  this  type  of  problem  annually.  Berner  (p630,  1984) 
indicates  “This  type  of  problems  may  be  pattern  recognition  tasks  with  a standard  solution,  or  they  may  be  ones  for 
which  a standard  algorithm  is  used”. 

2.  Level  two 

This  is  the  problem  that  exists,  but  remains  to  be  identified.  Once  it  is  identified,  the  solution  is  clear.  The 
instructor  might  know  what  is  wrong  in  the  problem  but  needs  to  face  the  situation  with  alternative  ways  for 
solving  the  problem.  That  is,  the  students  may  pick  problems  and  solutions  which  are  different  with  the 
instructor’s  expectation.  This  class  of  problems  involves  a sacrifice  or  a tradeoff  of  important  personal  and 
organizational  goals. 

These  two  types  of  problems  could  serve  for  learning  objectives  such  as  memorized  facts  or  known 
processes  They  present  a comparatively  easy  task  for  instructors  to  teach  and  for  students  to  master.  However,  unlike 
the  above  two  types  of  we  11 -structured  problems,  there  are  some  problems  that  can  not  be  solved  by  such  a simple 
way.  Problem  solvers  might  even  face  a tradeoff  situation  for  the  solutions  they  chose. 

Ill-structured  Problems 

3.  Level  three 

“The  problems  are  those  which,  even  if  the  diagnoses  are  known,  there  is  no  agreed-upon  method  of  solution” 
(Getzels,  1982).  Considered  strategies  may  each  have  their  associated  risks  and  benefits.  For  instance,  clients’ 
values  might  conflict  with  methods  that  agents  choose  although  these  methods  might  be  able  to  solve  clients’ 
problems.  This  class  of  problem  is  the  generalized  mess  that  is  so  complex.  Problem  solvers  need  to  go  through  a 
careful  diagnosis  process  for  decision  making.  Their  previous  experience  might  provide  better  help  than  the 
standard  answers  from  their  textbooks  (Berner,  1984).  It  is  the  type  of  problem  scenario  that  even  the  instructor  has 
difficulty  in  getting  a handle  on  what  the  problem  is. 

4.  Level  four 

This  type  of  problem  is  the  one  that  is  just  invented  or  conceived  recently.  And  a solution  may  or  may  not  be 
available  yet.  For  instance,  an  instructor  is  assigned  a new  project  from  his  department.  He  / she  asks  the  problem- 
solving groups  in  the  class  to  work  on  parts  of  this  project.  However,  the  situation  is  that  even  said  the  instructor 
himself  / herself  still  need  to  figure  out  how  to  complete  this  project. 

Instructors  and  facilitators  could  have  a better  idea  for  choosing  problems  based  on  the  above  framework, 
while  it  is  worth  to  know  that  what  is  unknown  for  a learner  may  be  known  for  others.  For  instance,  a level  2- 
problem  for  learner  A might  require  complicated  decision  analysis  that  is  akin  to  a level  3-problem,  while  for  learner 
B,  this  problem  might  be  like  only  a level  1 -problem. 


Conclusion 

Generally,  this  model  shows  not  only  the  complexity  of  problems  that  can  help  structure  the  instructor’s 
course  objectives  and  cultivate  students’  problem  solving  skills,  but  also  that  the  whole  environmental  structures 
should  be  considered  to  work  the  same  function.  For  instance,  for  naive  learners,  it  is  better  to  choose  the  Procedural 
SCM  Sequence,  the  Problem-stimulated  Format,  and  a well-structured  problem.  This  type  of  we  11 -structured 
environment  helps  non-experienced  students  to  become  familiar  with  the  basic  knowledge  of  subject  domain  and 
prepares  them  for  the  next  more  complex  problem  and  more  ill-structured  context.  In  conclusion,  learning  should  not 
just  be  considered  in  a systematic  format  but  should  also  be  weighed  in  a systemic  matter.  In  other  words,  PBL 
facilitators  should  realize  that  the  student  ability  used  to  solve  the  problem  is  dependent  on  the  problem’s  structure, 

O 

ERIC 


346 


326 


subject,  and  context.  In  this  decision  model,  a total  of  thirty-two  different  PBL  structures  are  arranged  from  simple  to 
complex.  Instructors  can  combine  different  levels  of  sequences  (four  subsequences),  formats  (Problem-stimulated 
and  Student-centered),  and  problem  structures  (four  levels)  to  conduct  the  PBL  context.  However,  this  decision 
model  should  just  work  as  a reference.  An  experienced  PBL  facilitator  should  decide  his  / her  own  instructional 
environment  after  considering  both  the  course  goals  and  students’  needs. 
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Abstract 

The  purpose  of  this  paper  is  to  present  the  results  of  the  implementation  of  an  instructional  approach  that 
supports  their  learning  of  instructional  design  (ID)  and  honors  teachers  ' beliefs  and  concerns  within  the 
ID  process.  We  present  the  theoretical  framework  that  supports  this  inquiry , summarize  how  instructional 
design  is  viewed  and  consequently  taught  ( the  designer's  view),  how  ID  is  used  by  teachers  (the  teacher's 
perspective ),  followed  by  a summary  of  collaboration  between  teachers  and  designers , and  teacher  inquiry 
into  their  teaching.  We  describe  our  research  method , including  our  instructional  approach.  We  summarize 
ways  that  teachers  used  to  describe  and  represent  their  practice  in  ID  projects , personal  ID  models , and 
course  evaluation , and  how  our  co-participatory  approach  helped  teachers  and  designers  to  learn  from 
each  other. 

Introduction 

Efforts  at  diffusing  instructional  design  (ID)  into  teaching  practice  have  largely  been  unsuccessful 
(Gustafson,  1993;  Martin  & Clemente,  1990).  Such  results  are  partly  due  to  the  different  worlds  that  characterize  the 
lives  of  designers  and  teachers.  One  means  of  diffusion  has  been  through  formal  ED  instruction.  However,  how  one 
teaches  instructional  design  comes  to  be  viewed  by  newcomers  as  the  means  by  which  it  ought  to  be  conducted  in 
practice.  Its  use  by  teachers  may  be  limited  if  the  process  is  represented  to  them  as  linear  and  overly  prescriptive,  or 
conducted  via  the  use  of  one  standard  model.  Instead,  ID  can  be  presented  as  a systematic  and  holistic  process  to 
examine  and  address  instructional  problems.  Teachers  can  be  introduced  to  a range  of  models  and  be  encouraged  to 
develop  their  own.  An  unrealized  potential  of  ED  instruction  is  to  use  it  as  a structured  means  to  help  teachers 
examine  their  teaching.  Thus,  ED  instruction  provides  an  important  opportunity  to  establish  meaningful  relationships 
between  teachers  and  designers. 

The  purpose  of  this  paper  is  to  explain  the  instructional  approach  and  present  the  results  of  using  that 
instructional  approach  that  includes  teachers’  beliefs  and  concerns  within  the  ED  process  itself  and  supports  their 
learning  of  instructional  design.  We  used  a co-participatory,  reflexive  approach  to  instruction,  in  which  we  viewed 
all  participants  as  learners  who  continually  reflected  and  appraised  our  joint  efforts  at  teaching  and  learning  ID. 
Along  with  this  stance  on  teaching,  we  used  the  ED  process  as  a tool  for  us  to  learn  about  these  teachers,  as  well  as  to 
guide  them  in  a self-examination  of  their  teaching  stance,  along  with  ED  learning.  Thus,  inquiry  was  built  into  the  ED 
process,  embracing  not  only  the  traditional  ED  components  of  needs  assessment  and  program  evaluation,  but  beliefs 
examination.  In  this  research,  our  inquiry  is  a study  of  our  own  teaching  of  instructional  design  with  in-service 
teachers. 

Theoretical  Perspective  On  Learning 

If  an  instructional  approach  is  to  empower  learners,  it  must  first  honor  the  learners’  role  and  agency  in  the 
construction  of  their  own  learning.  Three  basic  tenets  from  contemporary  learning  theory  inform  this  stance  and 
serve  as  the  foundation  of  our  instructional  approach.  We  begin  with  the  notion  that  learning  is  a constructive 
process  (e.g.,  Bartlett,  1932;  Bruner,  1990).  Knowledge  and  skills  are  developed  through  the  building,  linking,  and 
clarifying  of  the  personal  experiences  that  arise  as  learners  attempt  to  make  sense  of  their  worlds.  This  purposeful 
quest  toward  making  meaning  of  life’s  complexities  involves  multiple  components  that  comprise  the  learning 
enterprise:  culture,  cognition,  affect,  individual  differences,  and  so  forth.  Along  with  this,  we  recognize  that  learning 
is  developmental  and  historical  (John-Steiner,  1997;  Vygotsky,  1978).  Prior  learning  experiences  along  with 
available  strategies,  influence  how  learners  perceive  the  world  and  their  role  and  possibilities  within  it. 

Our  second  basic  tenet  holds  the  notion  that  learning  is  situated  and  mediated  in  social  contexts  (Brown, 
Collins  & Duguid,  1989;  Lave  & Wenger,  1991).  This  social  theory  of  learning  is  an  intersection  of  theories  of 
collectivity,  subjectivity,  power,  and  meaning  that  mediate  our  practice  and,  in  turn,  shape  our  identity  (Wenger, 
1998).  Cognitive  and  communicative  functions  that  are  inherent  in  a specific  social  world  are  intimately  entwined 
within  everyday  experiences  (I^ave  & Wenger,  1991).  Higher  mental  processes,  such  as  reflection,  have  their  origins 
in  the  community  of  learners  in  which  one  resides  (Moll,  1990).  As  Vygotsky  (1978)  has  written,  learning  is  first  at 
the  interpersonal  level,  but  that  capable  others  help  one  move  learning  to  an  intrapersonal  level.  Consequently,  it  is 
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critical  that  the  social  milieu  supports  intellectual  activity  in  order  for  its  participants  to  fully  realize  their  creative 
and  problem  solving  potential  (John -Steiner,  1997). 

Our  third  basic  tenet  requires  that  we  view  teaching  as  assisting  learners  (Tharp  & Gallimore,  1998).  Here 
we  mean  that  teachers  and  learners  engage  in  a co-participative,  responsive  learning  relationship  that  requires  a 
mutuality  that  transcends  the  traditional  and  authoritative  role  of  teacher  and  reactive  and  subordinate  role  of  student. 
The  construction  of  shared  understandings  through  authentic,  meaningful  activity  is  the  goal  of  the  enterprise.  This 
assistance  in  learning  of  ID  resonates  with  the  ideas  of  Rowland  and  colleagues  (e.g.,  Rowland,  Fixl  & Yung,  1992) 
that  instruction  consists  of  authentic  design  tasks,  modeling  of  design  expertise,  and  reflective  activities  (Schon, 
1987). 

Perspectives  On  Instructional  Design 

How  ID  is  taught  is  based  on  one’s  view  of  the  ID  process  and  can  be  regarded  as  the  “designer’s 
perspective.”  Researchers  have  configured  instructional  design  as  a systematic,  problem  solving  process  for 
designers  (e.g.,  Dick  & Carey,  1996;  Gagn6,  Briggs  & Wager,  1992).  ED  instruction  has  typically  involved 
presentation  of  these  models;  in  particular,  models  developed  for  novices  (e.g.,  Dick  & Carey,  1996)  depict 
prescriptive  processes  that  omit  contextual  features  (Edmonds,  Branch  & Mukherjee,  1994).  Attempts  to  develop  ID 
models  for  teachers  (Gerlach  & Ely,  1980;  Kemp,  Morrison  & Ross,  1996;  Reiser  & Dick,  1997;  Wedman  & 
Tessmer,  1990)  have  attempted  to  more  closely  approximate  teacher  tasks.  However,  systematic  processes  have 
proven  difficult  to  transfer  into  a teacher’s  daily  practice,  due  to  the  very  individualistic  and  personal  nature  of 
teaching,  the  complexity  of  classrooms,  and  the  multiple  influences  affecting  their  practice.  Because  teaching  is  a 
human  activity,  one  model,  especially  one  that  was  constructed  by  one  other  than  the  teacher  who  has  to  use  it,  is 
insufficient  to  be  responsive  to  teacher  needs.  Thus,  a process  that  represents  algorithmic  solutions  to  messy 
instructional  problems  is  ultimately  inappropriate  to  address  teacher  concerns. 

Meanwhile,  teachers’  views  of  the  ID  process,  the  teacher’s  perspective,  are  due,  in  part,  to  how  the  ED 
process  was  taught  to  them  and  their  experiences  using  the  process.  As  Martin  and  Clemente  (1990)  have  written, 
‘The  particular  decisions  we  make  as  we  teach  ISD  [Instructional  Systems  Development]  and  the  amount  of  time  or 
emphasis  we  give  to  aspects  of  ISD  ...affect  how  these  aspects  will  ultimately  be  used  in  the  classrooms”  (p.  83). 
Studies  have  identified  that  teachers  use  some  systematic  practices  (e.g.,  Branch,  1994;  Kennedy,  1994),  such  as 
learner  analysis,  goal  identification,  and  matching  tasks  and  assessment  to  specific  objectives.  Other  aspects  of 
systematic  practices  have  typically  not  been  addressed  by  teachers  in  these  studies,  including  the  lack  of  media 
selection  criteria,  needs  assessment  with  subject  matter  experts,  learner  characteristics,  and  matching  assessment  to 
overall  goals.  Studies  in  the  differences  between  new  and  more  experienced  teachers  (e.g.,  Driscoll,  Klein  & 
Sherman,  1994;  Rowland,  1991)  have  shown  that  novice  teachers  focus  on  immediate  tasks  and  objectives,  while 
more  experienced  teachers  take  a broader  view  of  instruction  and  goals.  In  these  studies,  teachers  have  cited  time  as 
a major  impediment  to  using  instructional  design,  that  “ID  is  too  time-consuming”  or  ‘There’s  not  enough  time  to 
use  the  ED  process.”  Instructional  design,  being  a complicated  process,  does  take  time  to  learn.  However,  this  initial 
view  can  jeopardize  a teacher’s  perception  and  valuation  of  the  process.  The  second  category  of  responses,  that 
“there’s  not  enough  time”  is  a broader  institutional  issue  involving  how  others  in  the  institution  view  the  teacher’s 
role  and  the  expert  outsider  and  a view  of  the  complexity  of  teaching  (e.g.,  Elbaz,  1983;  Jackson,  1968;  Lortie,  1975; 
Yinger,  1980).  We  maintain  that  the  relationships  between  teacher  and  designer  represent  another  category  of 
impediments  to  the  teacher’s  use  of  systematic  processes.  More  specifically,  how  one  views  relationships  needs  to  be 
examined.  That  is,  the  ways  that  teachers’  personal  practice  and  knowledge  about  teaching  are  honored  in  the 
learning  event  mediates  the  degree  to  which  they  consider  instructional  design  as  a tool  either  to  examine  or  improve 
their  teaching. 

Collaboration  Between  Teacher  And  Designer 

The  diffusion  of  innovation  in  any  institution  is  not  simply  a matter  of  implementing  that  innovation,  but, 
rather  is  based  on  the  interrelated  factors  of  the  features  of  the  innovation,  the  relationship  of  the  individual  with  the 
organization,  and  the  personal  characteristics  of  users  (Doktor,  Schultz  & Slevin,  1979).  The  diffusion  of 
instructional  design  into  schools  would  likewise  be  influenced  by  these  three  interrelated  factors.  A teacher  would 
assign  a value  to  the  innovation  relative  to  other  options  (Leonard-Barton,  1987).  How  one  teaches  ED  might  carry 
great  weight  in  how  a student  would  make  such  a value  decision,  relative  to  that  person’s  learning  experiences  and 
personal  and  professional  needs.  This  value  judgment  might  also  be  contingent  on  the  teacher’s  relationship  with  the 
institution;  in  particular,  the  support  and  reward  structures  in  place,  as  well  as  the  relationships  with  the  people  in 
that  institution.  Another  interconnected  influence  is  the  personal  characteristics  of  the  teachers.  One  set  of 
characteristics  could  include  years  of  experience,  professional  needs,  technical  competencies,  and  demographics 
(i.e.,  age  and  education).  A second  set  of  personal  characteristics  would  include  teachers’  views  towards  learning  and 
teaching,  and  dispositions  towards  themselves  as  learners  and  towards  outside  evaluators  or  experts. 
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The  applied  educational  research  enterprise  concerns  itself  with  practice  rather  than  theory  (LeCompte, 
1995),  and  collaboration  has  been  seen  by  evaluators  and  instructional  designers  as  a “cure  for  some  of  their  most 
difficult  investigatory  problems”  (p.  94).  Two  such  problems,  according  to  Huberman  (1991),  are  knowledge 
utilization  and  dissemination,  or  a “diffusion”  issue  (e.g.,  Gustafson,  1993).  Huberman  described  this  approach  as  the 
“two  communities”  problem,  with  differences  “in  norms,  rewards,  and  sensibilities  between  the  two”  (LeCompte,  p. 
95)  characteristics.  Huberman  characterized  three  variations  of  relationships  between  teacher-researcher 
communities  (i.e.,  teacher-designer).  The  first  relationship  is  “Hello-Goodbye,”  a short-term  involvement  in  which 
the  designer  shows  up  on  the  scene,  consults,  designs,  or  evaluates,  and  departs.  The  second  relationship  is  ‘Two 
Planets,”  in  which  the  relationship  is  an  interaction  of  different  cultures,  individuals  circling  around  each  other  but 
never  touching.  The  third  relationship  is  labeled  as  the  “Stand-Off,”  in  which  disagreement  between  the  parties  is  the 
rule.  Rather  than  facing  differences  and  issues,  the  parties  act  from  their  respective  positions  without  any  intent  to 
bring  out  differences  in  the  open,  to  learn  from  each  other’s  position,  or  to  re-examine  one’s  own  position. 

Rather  than  trying  to  increase  teachers’  knowledge  base  (i.e.,  utilization)  as  the  aim  of  professional 
development,  Rhine  (1998)  proposes  that  teachers’  engagement  with  research  should  be  stimulated  by  teachers  who 
desire  to  inquire  into  student  learning.  “Our  human,  bounded  rationality  dictates  the  value  of  educational  research  to 
the  teaching  community  is  not  the  acquiring  of  research-based  knowledge  of  students’  understanding,  but  the  process 
of  teachers  engaging  with  that  knowledge  and  considering  implications  for  their  instruction”  (p.  27).  His  reasoning  is 
that  teachers  cannot  know  and  process  everything  there  is  to  know  (bounded  rationality).  The  aim,  according  to 
Rhine  (1998),  is  to  shift  teachers’  focus  from  teaching  to  inquiring  into  learner’s  thinking,  although  we  view  the  shift 
as  away  from  a view  of  teaching  as  management  of  students  to  ways  in  which  teachers  support  students’  learning. 
Rhine  suggests  that  teachers  need  to  be  provided  with  guidelines  on  using  research-based  resources  and  provide 
access  to  these  resources  using  “teacher-friendly”  language  and  formats.  Collaboration  then  requires  the  “building  of 
relationships  between  research-based  models  of  children’s  thinking,  their  own  student’s  thinking,  and  how  they  can 
interpret  the  model  in  light  of  their  own  students  and  classrooms”  (p.  29). 

One  way  for  teachers  and  instructional  designers  to  consider  each  other’s  practices  is  to  report  the  results  of 
teacher  planning/ED  processes  in  publications  each  audience  might  read  (Earle,  1994).  Another  forum  for 
relationship  building  is  through  formal  instruction,  including  teacher  education,  as  well  as  graduate  programs.  A 
third  possibility  involves  teachers  and  outside  ED  practitioners  who  might  be  conducting  research  in  schools. 
Instructional  designers  should  be  encouraged  to  discover  what’s  happening  in  schools  and  attempt  to  understand  the 
lives  of  teachers,  their  clients. 

Teacher  Inquiry 

The  idea  of  teacher-as-researcher  has  been  based  on  the  idea  that  the  teacher,  who  is  at  the  forefront  of  what 
happens  in  classrooms,  can  contribute  to  knowledge  building  on  teaching.  This  knowledge  informs  not  only  the 
teacher,  but  also  can  contribute  to  the  educational  community  (Lytle  & Cochran-Smith,  1990).  Furthermore,  teachers 
as  researchers  come  to  learn  about  and  debate  the  research  enterprise,  about  “what  counts  as  new  knowledge”  and 
how  one  comes  to  arrive  at  this  new  knowledge.  One  way  of  implementing  research  into  teaching  lives  has  been 
action  research  (e.g.,  Duffield  & Robinson,  1998;  Keating,  Diaz-Greenburg,  Baldwin  & Thousand,  1998;  Somekh, 
1997).  Inquiry  as  action  research  adds  data  collection  and  analysis  to  the  teacher’s  routine  to  help  them  become  more 
objective  in  interpreting  their  experiences.  Rather  than  making  judgments,  teachers  use  data  to  help  them  describe 
what  is  occurring  and  to  provide  a basis  for  action. 

Reflective  practices  are  another  aim  of  action  research.  Reflection  is  an  “active  process  providing  an 
opportunity  to  look  at  past  experiences  and  relate  them  to  future  action”  (Keating  et  al,  p.  383).  Winn  (1989) 
suggests  a reflective  practicum,  in  which  Schon’s  (1987)  reflection-about-action  and  reflection-in-action  can  be 
supported.  Technical  competencies  are  not  enough  (Quinn,  1994),  that  a creative  supplement  is  also  necessary 
(Rowland,  Parra  & Basnet,  1994),  one  that  encourages  flexibility  and  intuition  based  on  experience  (Nelson, 
Magliaro  & Sherman,  1988),  to  address  human  instructional  problems. 

We  propose  the  use  of  instructional  design  (ID)  as  a systematic,  intellectual  tool  to  help  teachers’  to  “relate” 
past  experiences  to  future  action  by  (1)  examining  their  learning  beliefs  and  instructional  practices  and  (2)  designing 
responsive  approaches  to  instructional  problems.  Thus,  we  view  teachers  within  the  ED  process  itself,  structurally 
supporting  teacher  self-examination  (inquiry),  but  also  as  a tool  (technology)  to  systematically  examine  instructional 
problems  and  propose  responsive,  bounded  solutions.  Our  view  is  that  the  tools  of  analysis,  design,  and  evaluation 
within  the  ED  process  can  help  teachers  to  realize  the  differences  and  complexity  of  student’s  thinking.  Rather  than 
closing  down  the  inquiry,  or  bounding  the  problem  prematurely,  our  view  of  ED  is  to  examine  the  possibilities 
inherent  in  any  instructional  problem.  The  value  of  the  systems  approach  is  “taking  in  the  whole  problem,”  while 
examining  the  component  issues.  Our  view  of  ED  for  teacher  use  is  that  the  process  retains  the  bounded  rationality  of 
professionals,  but  ED’s  systematic  features  provide  structure  to  keep  learning  in  the  forefront  and  continue  to 
examine,  design,  and  evaluate  one’s  efforts. 
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Research  Questions 

Our  research  orientation  included  all  participants,  teachers  and  instructors.  Thus,  the  research  questions  for 
this  study  included  both. 

1 . How  do  teachers  view  teaching  in  what  they  designed? 

2.  How  do  teachers  view  teaching  in  what  they  represented? 

3.  How  do  teachers  talk  about  their  teaching? 

4.  In  what  ways  does  our  co-participatory  approach  help  teachers  and  designers  (i.e.,  instructors)  learn  from  each  other? 


Method 

Participants  And  Setting 

Participants  included  two  instructors  and  23  practicing  teachers  enrolled  in  an  instructional  design  course, 
as  part  of  a master’s  program  sponsored  jointly  by  their  school  system  and  a state  university.  The  K-12  teachers’ 
instructional  responsibilities  ranged  across  all  disciplines  and  contexts  (e.g.,  general  education,  special  education, 
vocational-technical  education,  consulting  teacher,  and  librarian).  The  focus  of  this  master’s  program  was  the 
integration  of  instructional  technology,  and  the  curriculum  was  designed  so  that  all  coursework  should  be  interrelated 
and  projects  emanate  from  the  teachers’  daily  practice.  The  ID  course  (the  second  of  a 10-course/30  credit-hour 
program)  was  offered  in  one  of  the  county  schools  and  met  for  three  hours  each  week  over  a 15- week  semester. 

Instructional  Approach 

As  instructors  we  were  responsible  for  introducing  these  teachers  to  the  formal  instructional  design  process, 
and  as  such,  our  voices  valued  or  signified  this  process  by  the  way  in  which  we  advocated  and  modeled  the  ID 
process,  using  it  to  study  our  own  teaching.  Although  we  presented  a traditional  treatment  of  the  process  (i.e., 
analysis,  design,  evaluation),  our  instructions  to  the  teachers  were  to  use  this  framework  as  a tool  to  examine  and 
bring  forth  for  examination  the  process  they  use  to  guide  their  practice.  Rather  than  depicting  instructional  design  as 
a complex  algorithm,  we  represented  ID  as  a systematic  means  to  examine  important  instructional  issues  before 
settling  on  a response.  Instead  of  specifying  a set  of  rules  or  procedures,  which  could  not  cover  all  contingencies  of 
human  learning,  we  advocated  to  teachers  that  they  consider  a range  of  possibilities  that  might  responsively  address 
the  learning  needs  of  their  students  (Magliaro  & Shambaugh,  1997).  We  engaged  them  in  tasks  that  would  help  them 
to  try  components,  reflect  on  their  appropriateness,  and  make  revisions  based  on  this  reflection  and  our  feedback. 

Our  reflexive  approach  to  teaching  instructional  design  views  instructor  and  student  as  co-participatory 
learners,  in  which  participants  learn  from  each  other  (Shambaugh  & Magliaro,  1995).  As  such,  all  participants  are 
inquiring  into  their  teaching  practices,  including  the  instructors  for  the  course.  Through  constant  dialogue  (i.e.,  live 
communication,  computer-mediated-communication),  we  all  design  with  each  other  and  serve  as  formative 
evaluators  in  a supportive  and  collegial  way.  Logistieally,  we  create  the  participation  structures,  including  classroom 
activities,  learning  tasks,  electronic  mail,  individual  conferences,  and  text  (Shambaugh  & Magliaro,  1997),  in  which 
dialogue  and  performance  are  supported.  The  teachers  bring  their  instructional  design  problems  to  the  table  and 
mediate  the  participation  structures  in  order  to  address  their  design  problem,  which  may  have  been  a part  of  their 
overall  professional  goals. 

Being  reflective  and  critical  together  required  an  element  of  trust  and  history  within  the  cohort.  The  ID 
class  was  preceded  by  a course  in  educational  psychology,  taught  by  the  same  instructor,  in  which  the  teachers 
examined  learning  theories  (i.e.,  behaviorism,  cognitive  psychology,  and  social  constructivism)  and  wrote  papers  that 
juxtaposed  these  approaches  to  their  teaching.  These  activities  enabled  the  teachers  to  name  the  activities  and 
principles  that  they  had  been  using  in  the  classroom.  No  one  theory  was  deemed  “better”  than  the  others.  Teachers 
were  encouraged  to  adopt,  adapt,  or  create  a personal  theory  of  learning  that  could  be  articulated  to  another 
interested  party  (e.g.,  colleague,  parent,  friend,  etc.).  A final  reflection  paper  on  their  learning  beliefs  and 
corresponding  instructional  practices  served  as  a transition  to  the  instructional  design  course,  in  which  this  self- 
examination  mode  continued. 

The  sequence  for  the  15- week  semester  course  included  three  weeks  in  which  the  context  for  instructional 
design  was  established.  Part  of  this  context  was  to  have  these  teachers  continue  to  examine  their  beliefs  about 
learning  and  teaching,  as  these  became  the  foundation  forjudging  the  veracity  and  cohesion  of  their  ID  project. 
Teachers  were  asked  to  first  represent  their  own  design  or  planning  process.  This  served  two  purposes:  (1)  to  honor 
their  present  conceptions  of  the  ID  process,  and  (2)  to  provide  a tangible  representation  of  a tacit  process  that  could 
be  examined  in  a more  open  manner.  Throughout  their  se mester,  we  asked  them  to  update  their  models  based  on  any 
new  ideas,  revelations,  or  understandings  that  emerged  from  either  class  or  their  project  work. 

The  core  of  the  ID  course  content  consisted  of  a 9-phase  instructional  design  sequence,  with  the  principal 
assessment  method  involving  ongoing  drafts  and  a completed  ID  project.  A mission  statement  task,  the  first  phase  of 
the  sequence,  provided  a written  means  to* evaluate  the  extent  to  which  personal  beliefs  were  incorporated  in 
individual  projects.  Across  the  semester,  we  provided  overviews  and  group  activities  examining  important  ID 


process  components,  including  analysis  (i.e.,  needs  assessment),  design  (lesson  sequence,  assessment,  instructional 
framework,  instructional  media,  and  prototype  lesson),  and  program  evaluation.  Teachers  were  introduced  to  a range 
of  ID  tools,  including  ID  models  and  taxonomies.  Teachers  formed  work  groups  in  which  they  either  all  worked  on 
the  same  project  or  worked  on  complementary  projects.  Class  time  from  each  class  period  was  devoted  to  group 
work  sessions.  During  the  final  week  we  returned  to  a self-examination  in  which  teachers  revised  and  shared  their 
personal  ID  models  and  wrote  a self-evaluation  of  their  learning  and  their  responses  to  the  course. 

Data  Sources  And  Analysis 

Three  domains  of  analysis  were  identified  and  explored:  we  analyzed  what  teachers  designed  in  their  ID 
projects,  how  teachers  represented  their  thinking  about  and  planning  processes  in  their  ID  models,  and  what  teachers 
said  about  their  teaching  after  the  course.  Data  sources  included  teachers*  ID  projects,  their  personalized  ID  models, 
and  self-reports  on  their  learning  from  a course  evaluation. 

A recursive  process  of  categorization  and  theme  building  were  used  to  investigate  our  work  (e.g.,  Spradley, 
1980).  Thirteen  completed  individual  or  group  projects  were  available  for  analysis.  Analysis  consisted  of  the  extent 
to  which  teachers  conducted  research  to  inform  their  understanding  of  their  instructional  problem,  the  extent  to 
which  teachers’  learning  beliefs  were  consistently  and  coherently  applied  throughout  the  project,  and  to  what  degree 
their  “technology  component”  supported  their  project.  Research  efforts  were  examined  in  the  project’s  needs 
assessment.  The  application  of  learning  beliefs  was  analyzed  by  comparing  what  teachers  wrote  in  their  mission 
statement  with  their  design  components.  The  technology  component  was  examined  in  the  project’s  instructional 
media  section.  Teachers  were  asked  to  represent  graphically  and/or  narratively  the  components  and  processes  they 
considered  in  their  everyday  practice.  Nineteen  personal  ID  models  were  analyzed  by  describing  the  features  of  the 
visuals  metaphor  and  the  language  from  a written  narrative  accompanying  the  visual.  Initially,  each  model  was  given 
a label  that  attempted  to  encapsulate  that  teacher’s  view  of  teaching.  A second  step  involved  assigning  these  labels  to 
categories  with  similar  labels.  The  teachers  also  provided  feedback  on  the  impact  of  the  course  and  master’s  program 
on  their  professional  lives  in  an  end-of-the-course  questionnaire.  Teacher  responses  were  coded  as  to  themes  and 
summarized  into  categories. 

Results 

What  Teachers  Designed:  Their  ID  Projects 

All  of  the  projects  were  based  on  personal/professional  beliefs  about  what  it  meant  to  teach  in  particular 
content  areas  (e.g.,  spelling,  science,  video  as  a persuasive  medium,  calculator  use,  aerospace,  geography).  None  of 
the  projects  brought  forward  research  findings  from  texts  assigned  from  the  program’s  previous  educational 
psychology  course  (Bruer,  1993;  Schauble  & Glaser,  1996),  which  examined  the  complexity  of  content  to  be  learned 
and  learning  environments  that  supported  this  learning. 

Nine  of  the  13  projects  achieved  a consistency  of  teachers’  learning  beliefs,  expressed  through  a mission 
statement,  across  their  project’s  design  components.  Four  projects  did  not  demonstrate  a consistent  stance  between 
mission  and  instructional  action.  For  example,  one  mission  statement  expressed  a desire  to  “assist  learners  to  reach 
their  goals  through  collaborative  partnerships”  but  did  not  examine  what  “assistance”  or  what  “collaborative 
partnerships”  meant  in  his  teaching.  Another  mission  statement  advocated  “active  involvement... using  a variety  of 
instructional  methods... to  produce  self-directed  learners,”  although  the  ID  document  only  described  direct 
instruction  and  strictly  adhered  to  Virginia’s  Standards  of  Learning. 

Achieving  consistency  of  beliefs  across  components  was  not  always  responsive  to  learners’  needs.  One 
teacher  cited  being  “practical  and  hands-on”  in  his  mission  statement,  which  was  exhibited  in  activities,  but  was 
inadequate  to  support  the  conceptual  learning  called  for  in  the  project.  Materials  in  the  lesson  prototype  did  not 
provide  any  explanation  for  the  conceptual  content  underlying  the  activities.  The  teacher  did  not  conduct  research  on 
ways  to  teach  the  content  or  re-evaluate  or  describe  the  pre-existing  curriculum  materials  that  were  inserted  into  the 
project.  Another  student  wrote  about  “collaborative  partnerships”  but  did  not  specify  any  affective  dimensions  in 
project  goals  to  support  these.  In  another  project,  in  which  cooperative  groups  were  stressed,  the  assessment  plan 
accounted  for  skill  learning  without  any  provisions  for  social  learning  goals.  Coherence  across  design  components 
was  found  in  eleven  of  the  thirteen  projects.  In  all  eleven  instances  of  coherence,  lessons  identified  goals,  which  had 
been  determined  from  the  needs  assessment.  Of  the  two  projects  that  were  judged  not  to  have  coherence,  a 6-week 
lesson  outline  did  not  match  activities  with  goals,  and  a geography  unit  specified  goals  in  a mission  statement  but 
were  not  identified  in  later  design  components,  although  lesson  activities  were  keyed  to  state  standards.  Six  of  the 
thirteen  projects  used  Virginia’s  Standards  of  Learning  (SOL’s)  as  goals.  Three  of  the  thirteen  projects  used  an 
activities  sequence  to  address  their  instructional  framework,  assessment,  and  media.  Four  of  the  projects  minimally 
described  their  proposed  instructional  framework. 

A technology  component  was  required  in  their  ID  projects,  although  only  the  instructional  design  issues  of 
media  selection  and  use  were  to  be  specified.  Although  this  technology  component  was  a major  influence  in  the 
selection  of  an  instructional  problem,  teachers  identified  a full  range  of  instructional  media.  Chosen  media-supported 
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projects  that  addressed  current  teaching  (e.g.,  audio  spellchecker,  LEGO’s™,  maps,  floral  supplies),  were  mandated 
for  future  courses  (i.e.,  graphing  calculators),  or  were  innovations  (programmable  logic  controller,  orientation 
video).  No  research  was  conducted  on  how  these  media  choices  supported  learning.  Instructional  media  rationales 
were  frequently  specified  within  lessons,  but  without  many  details  on  their  use. 

What  Teachers  Represented:  Their  Personal  ID  Models 

Teacher’s  ID  models  were  examined  along  two  dimensions:  the  metaphors  they  constructed  and  their 
location  in  the  ID  process.  Nineteen  representations  of  instructional  design  included  visual  metaphors  that  included  a 
circular  cycle,  signpost,  flower  (2),  flower  garden,  kite,  dart  board,  balancing  scales,  pizza  parlor,  billiards, 
passenger  train  (2),  performance  stage,  triangular  puzzle  pieces,  fish,  baseball  diamond,  steps,  and  a story.  Teachers’ 
views  of  instructional  design  were  grouped  into  four  categories.  The  first  category,  accounting  for  four  teachers, 
viewed  ID  as  a “sequence,”  and  included  the  two  train  models,  a continuous  cycle,  and  a four-step  model.  The 
second  category  grouped  three  teachers’  “holistic”  views  of  ID  and  included  three  teacher  models  with  flowers  as 
themes.  One  of  these  teachers  used  a flower  to  represent  ID  as  a complex,  dynamic,  and  systematic  process, 
representing  feedback,  review,  and  revision  with  double-pointed  arrows  and  intersecting  lines.  As  one  must  be 
attentive  to  a flower’s  health,  “good  ID  must  be  responsiveness  to  the  learner’s  health.”  Another  flower  model 
equated  an  ED  task  with  a garden  design  task,  representing  needs  assessment  as  a plot  plan  and  that  an  analysis  of  the 
parts  as  they  relate  to  the  whole  plot  was  necessary.  The  framework  to  the  plot  was  viewed  as  learning  principles; 
however,  the  teacher  viewed  this  framework  as  one  to  “plan,  manage,  and  organize  information.” 

The  third  category  of  how  nine  teachers  viewed  ED  was  labeled  “process”.  Examples  included  a fish 
“eating”  knowledge,  sinking  billiard  balls,  and  hitting  the  curriculum  “bulls-eye”;  in  this  case,  state  learning 
standards,  with  a dart  representing  teaching  approaches.  Other  activities  included  acting  out  roles  on  a stage, 

“running  the  bases”  in  baseball,  writing  a story,  and  running  a pizza  parlor.  A fourth  category,  “balance,”  was  used  to 
characterize  three  teachers,  including  two  special  education  teachers  who  viewed  their  daily  teacher  lives  as 
unpredictable.  One  teacher  used  signposts  to  represent  the  decisions  to  turn  left  or  right.  ‘Turning  left”  meant  facing 
issues  she  had  no  control  over,  including  the  demands  of  curriculum,  teacher  expectations,  and  state  requirements, 
while  “turning  right”  meant  working  with  student  strengths  and  weaknesses.  The  second  special  education  teacher 
saw  ED  as  a means  to  balance  learner  needs  and  teacher  expectations. 

Teachers’  ED  models  were  also  categorized  by  how  they  viewed  teaching  and  their  location  in  the  process. 
The  first  view  of  teaching  was  “disseminating  knowledge,”  and  included  three  models  concerned  with  teaching 
practical  skills  (horticulture),  knowledge  (social  studies),  and  meeting  objectives  (technology  education).  A second 
view  of  teaching  was  “addressing  learner  needs”  and  accounted  for  four  of  the  19  models.  Models  and  narrative  that 
matched  this  label  included  “growing  a squash  garden,”  “attending  to  a flower’s  health,”  designing  a garden,  and 
playing  with  puzzle  pieces.  A third  category  viewed  teaching  as  “addressing  external  expectations,”  accounting  for 
six  models.  These  models  included  the  teacher  as  cue  ball;  rounding  the  bases  on  a baseball  field;  being  pulled  by  a 
train  (the  state’s  standards  of  learning);  and  sinking  the  striped  billiard  balls  (instructional  issues)  avoiding  the  solid 
balls  representing  time,  state  standards,  and  interruptions  (the  “eight-ball”).  The  fourth  category  included  the  two 
special  education  teachers  who  viewed  themselves  as  reactive,  always  in  the  middle,  making  decisions  and  balancing 
expectations  for  students  with  expectations  of  other  teachers,  parents,  and  the  system. 

What  Teachers  Said  About  Their  Learning 

At  the  end  of  the  course,  teachers  were  asked  to  reflect  on  the  impact  of  the  program  on  their  professional 
and  personal  lives.  The  possibility  of  teasing  out  the  impact  of  just  the  ED  experience  was  deemed  counterproductive 
given  the  seamless  nature  of  the  program.  All  20  respondents  reported  that  the  activities  made  them  more  aware  of 
their  teaching,  with  8 stated  in  an  active  way  that  they  are  agents  of  change  (e.g.,  “I  am  more  conscious  about  the 
choices  that  I make  for  instruction.”).  Four  respondents  reported  a better  understanding  of  the  students  within  their 
design  decisions.  Eighteen  teachers  reported  ways  that  the  program  had  affected  their  students.  While  13  infused 
ideas  from  the  program  indirectly,  5 reported  that  they  showed  their  students  their  assignments,  shared  their  struggles 
with  papers,  due  dates,  etc.  and  so  forth.  One  stated  that,  ‘They  like  knowing  I’m  also  a student  - modeling  life-long 
learning.” 

From  a personal  perspective,  10  recognized  the  time,  financial,  and  energy  commitment  of  engaging  in 
reflective  professional  development.  One  found  the  program  to  be  a social  outlet.  Another  reported  a newly  found 
self-efficacy  as  a learner  in  a demanding  intellectual  endeavor. 

Across  their  comments,  the  teachers  commented  about  the  revived  respect  they  had  for  their  own 
profession.  One  stated  that,  “My  commitment  to  my  job  has  increased.  I really  know  why  I do  what  I do!”  These 
were  the  initial  signs  of  individuals  willing  to  critically  examine  their  own  teaching  and  gaining  the  confidence  to 
share  their  voices. 
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Conclusions 


Teachers  cited  a number  of  benefits  of  the  course  for  their  teaching:  looking  at  planning  in  an  in-depth 
fashion  and  creating  something  they  could  use  in  the  classroom,  improvements  in  communication,  collaboration, 
problem  solving,  and  creating  thinking  skills.  In  terms  of  instructional  design,  one  student  remarked  having  a “much 
greater  appreciation  of  the  design  process  now.”  Teachers  cited  “different  ways  to  think  about  the  learners”  and 
“forces  the  teacher  to  look  at  lots  of  details  to  designing  instruction  and  curriculum.” 

What  these  teachers  designed  and  represented  was  also  a product  of  the  political  and  social  constraints  of 
school  environment  and  their  personal  lives.  Finding  themselves  in  the  role  of  student  was  a new  experience  for  some 
of  these  teachers.  Juggling  a new  set  of  demands  on  top  of  their  already  busy  teaching  lives  challenged  all  of  the 
teachers.  Reflective  activities  that  asked  teachers  to  examine  their  teaching  and  learn  a new  process  that  scrutinized 
their  teaching  was  for  some  a difficult  experience.  Even  with  experienced  special  education  teachers  who  wrote  goals 
every  day,  the  question  “What  is  a goal?”  was  frequently  mentioned.  One  of  us  remarked,  “We  really  put  a lot  of 
stumbling  blocks  in  their  way.  They  have  to  start  thinking  about  it  and  it's  now  becoming  really  awkward  for  them. 
We're  interrupting  the  flow  of  what  they  have  and  when  you  do  that,  they’re  perplexed.” 

We  found  that  all  of  the  ID  projects  relied  on  personal  beliefs  on  ways  to  teach  their  content  and  to  support 
their  design  decisions,  including  their  instructional  technology  component.  Furthermore,  none  of  the  reading  done 
from  a previous  educational  psychology  course  was  cited  in  the  projects.  However,  this  was  not  a surprise  as  some  of 
the  teachers  in  the  previous  class  struggled  to  see  the  connection  between  research  and  practice,  and  publicly 
dismissed  the  research  as  not  relevant  to  their  needs.  While  the  teachers  sometimes  found  it  difficult  to  reflect  on  and 
write  about  their  teaching,  even  representing  their  teaching  with  a model,  moving  to  the  next  step  and  making 
changes  in  their  practice  is  even  more  difficult  considering  the  professional  and  political  challenges  they  face. 

The  personal  ID  model  task  was  structured  to  encourage  teachers  to  represent  their  own  model  of  ID 
components,  rather  than  imposing  ours  or  someone  else’s.  The  value  of  visual  representations  for  teachers  is,  as 
Elbaz  (1983),  cited  in  her  study  of  teacher  thinking,  to  invite,  rather  than  compel,  conformity.  ID  is  known  for  its 
models  on  how  instruction  should  be  constructed,  and  we  believe  that  teachers  can  learn  from  these  models,  but  what 
is  more  productive  is  encouraging  teachers  to  develop  their  own.  Because  the  course  was  a first  experience  with  the 
process,  teacher  understanding  of  the  whole  process  and  the  relationship  of  the  components  to  each  other  were  not 
always  understood  and  represented  in  their  models.  Teachers’  representations  of  instructional  design  tended  to  adopt 
aspects  of  our  representation  of  the  process  and  what  we  valued.  However,  their  models  used  personal  metaphors 
that  illuminated  not  only  instructional  design  components  but  also  views  of  their  teaching,  as  well  as  their  views  of 
students  and  the  school  environment.  Metaphors  provided  teachers  with  a means  for  “clustering  images  for 
exploration  and  analysis  of  teacher  thinking”  (Bullough  & Gitlin,  1995,  p.  66).  Metaphors  within  these  models 
helped  teachers  to  explore  who  they  think  they  are,  although  it  was  not  clear  that  to  us  that  they  were  aware  of  the 
significance  of  these  representations.  Several  of  the  teachers  who  viewed  their  teaching  lives  as  balancing  the 
expectations  of  students  and  the  expectations  of  other  teachers  or  schooling  may  not  be  aware  of  their  loss  of  voice 
due  to  their  perceptions  of  their  positions  or  their  preferences  for  taking  responsibility. 

One  limitation  from  this  brief  research  summary  was  not  being  able  to  describe  in  more  detail  the  contexts 
in  which  these  teachers  led  their  teaching  lives,  for  it  is  out  of  these  contexts  which  give  rise  to  teacher  voices 
(Hargreaves,  1996).  On  the  other  hand,  we  have  been  conscious  not  to  generalize  or  romanticize  what  teachers 
designed,  represented,  and  said  into  one  teacher’s  voice,  a danger  in  some  educational  research  (Hargreaves,  1996). 
In  this  study  we  tried  to  include  data  that  represented  different  aspects  of  the  teachers’  voices,  including  what 
teachers  designed  and  represented.  However,  the  use  of  representation  needs  its  own  context  to  more  accurately 
characterize  its  use.  As  one  teacher  remarked  from  Bullough  and  Gitlin’s  (1995)  use  of  written  metaphor  for  teacher 
examination,  “I  think  the  metaphor  is  actually  restrictive  in  describing  ourselves.  Maybe  a multiple  metaphor”  (p. 

71).  Although  we  have  not  conducted  a careful  study  of  teachers’  ID  models  with  the  teachers,  the  task  appears 
useful  for  the  next  steps  in  enabling  reflection  and  voice.  Moreover,  writing  school  history,  based  on  Bullough  and 
Gitlin’s  (1995)  guidelines,  might  be  a useful  technique  for  cohort  in-service  programs  to  continue  teacher 
examination  and  to  contextualize  themes  that  might  emerge  from  a collaborative  analysis  of  this  history. 

Significance 

Our  approach  to  teaching  instructional  design  consisted  of  examining  a teacher’s  basis  for  action,  one’s 
learning  beliefs,  as  a key  component  of  the  design  process,  as  well  as  a systematic  use  of  the  ID  process  to  analyze, 
design,  and  evaluate  responsive  approaches  to  instructional  problems.  Initially,  some  teachers  viewed  instructional 
design  as  a complex  algorithmic  procedure.  One  request  from  a teacher  was  illuminating  and  represents  one  of 
instructional  design’s  most  mis-characterized  issues:  ‘Tell  how  you  can  condense  instructional  design  and  still  be  as 
effective.”  We  admit  that  a systematic  examination  of  instruction  requires  time  and  effort;  however,  as  cited  by  one 
teacher  at  the  end  of  the  course,  “I  now  have  different  ways  to  think  about  learners,”  suggests  that  the  effort  is  worth 
it.  Instructional  design’s  potential  has  gone  unrealized  by  its  depiction  as  merely  a procedural  tool  for  teachers. 
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Rather,  we  advocate  ID  as  a thinking  tool,  one  that  supports  systematic  teacher  re-examination  and  complements 
existing  teacher  thinking  and  practices.  The  ID  course  provides  teachers  with  the  time,  while  the  ID  process  provides 
teachers  with  tools  for  reflection  on  their  teaching  (Wildman  & Niles,  1987).  The  course  provides  teachers  with  a 
forum  to  talk  openly  about  the  environment  in  which  they  work  in  and  how  this  context  may  limit  the  kind  of 
teaching  they  want  to  practice.  Thus,  what  is  a pragmatic  process  for  action,  also  represents  a tool  for  critical  inquiry. 

Another  important  benefit  from  our  reflexive  approach  is  its  co-participatory  nature,  in  which  both 
instructor  and  student  are  considered  as  learners  and  both  continually  strive  to  learn  from  each  other  and  to 
continually  examine  our  teaching  and  learning,  and  to  honor  participant  thinking  and  activity.  Our  efforts  have  been 
to  improve  communication  with  and  between  teachers,  learn  from  teachers,  set  the  stage  for  sustained  teacher 
reflection  and  examination,  and  develop  habits  of  systematic  inquiry. 

ID  Instruction  represents  a significant  challenge  to  anyone  who  has  taught  it,  owing  to  the  range  of 
systematic  components  that  comprise  ID  and  the  investment  of  time  and  attention  needed  to  support  student  learning, 
particular  with  authentic  design  activities.  If  the  view  of  teachers  is  that  of  reflective  practitioners  (Schon,  1983), 
then  the  beliefs  and  concerns  of  teachers  need  to  be  incorporated  somehow  within  the  ID  process,  as  well  as  within 
ID  instruction.  Instructional  design  can  be  presented  to  teachers  in  ways  that  honor  their  beliefs  and  experiences, 
prompts  a re-examination  of  their  teaching,  and  encourages  heuristic  responses  to  instructional  problems.  Thus,  ID 
instruction  represents  not  only  a significant  teaching  challenge,  but  a significant  opportunity  to  help  teachers  re- 
examine their  teaching  and  for  teachers  and  designers  to  learn  from  each  other. 
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Abstract 

This  study  first  examined  the  motivation  construct  with  personal  goals , agency  beliefs , and  emotions  in  a 
web  assisted  learning  environment.  It  also  investigated  the  influence  of  personal  goals,  agency  beliefs,  and 
emotions  on  the  students'  learning  achievement  from  fundamental  biology  courses  assisted  with  web  pages. 
Finally,  it  examined  the  relationships  among  personal  goals,  agency  beliefs , and  emotions.  A survey 
questionnaire  was  designed  to  measure  goals,  agency  beliefs,  and  emotions  (Appendix  A).  Learning 
achievement  was  measured  by  the  course  grades. 

Three  research  questions  were  addressed:  (1)  Are  the  responses  to  the  questionnaire  for  measuring 
personal  goals,  agency  beliefs,  and  emotions  consistent  with  the  three-component  motivation  model 
proposed  by  Ford?  (2)  Do  students'  personal  goals,  agency  beliefs,  and  the  emotions  of  using  the  web 
pages  relate  to  their  learning  achievement?  (3)  Do  students'  personal  goals,  agency  beliefs,  and  emotions 
relate  to  each  other? 

Three  hundred  and  fifty  six  students  have  participated  in  the  study.  These  students  have  taken 
undergraduate  fundamental  biology  classes  in  which  the  instructors  used  web  pages  as  supplemental  tools 
to  assist  student  learning.  After  the  final  exam,  the  students  filled  out  an  on-line  questionnaire  in  order  to 
measure  their  personal  goals,  agency  beliefs,  and  emotions  in  using  web  pages  to  assist  learning.  Students' 
course  grades  were  used  to  measure  their  learning  achievement. 

A confirmatory  factor  analysis  with  Lisrel  VI  was  used  to  examine  the  construct  validity  of  the 
questionnaire  (Question  I).  Cronbach  Alpha  reliability  values  on  personal  goals,  agency  beliefs,  and 
emotions  were  calculated  with  SPSS  8.0  to  indicate  the  internal  consistency  of  the  questionnaire  (Question 
1).  The  causal  paths  among  the  personal  goals,  agency  beliefs,  emotions,  and  learning  achievement  were 
examined  using  structural  equation  modeling  with  Lisrel  VI  (Question  2 and  Question  3). 

The  Cronbach  alpha  reliability  coefficients  on  12  selected  items  of  the  motivation  instrument  for  students' 
personal  goals,  agency  beliefs,  and  emotions  were  0.84,  0.85,  and  0.89,  respectively.  The  confirmatory 
factory  analysis  on  12  selected  items  of  the  motivation  instrument  showed  a fairly  good  fit  of  the  theoretical 
model  of  motivation  proposed  by  Ford  (1992).  These  indicated  that  the  motivation  instrument  with  12 
selected  items  has  moderate  reliability  and  fairly  good  construct  validity. 

The  data  from  the  structural  equation  modeling  analysis  supported  that  there  were  significant  and  direct 
causal  links  from  students'  personal  goals  to  their  course  grades,  from  personal  goals  to  emotions,  and 
from  emotions  to  personal  agency  beliefs. 

Introduction 

Classroom  use  of  the  World  Wide  Web  (WWW)  is  seen  as  a new  learning  tool  to  reach  students  (Quinlan, 
1996).  The  Computer  Center  at  a northeastern  university  has  initiated  a web-based  learning  resource,  the  Virtual 
Classroom.  The  Virtual  Classroom  is  a listing  of  courses  that  use  web  pages  as  a supplemental  tool  to  enhance 
students’  learning.  Instructors  who  join  the  Virtual  Classroom  have  created  their  own  instructional  web  pages.  The 
pages  usually  include  class  syllabi,  practice  exams,  and  related  instructional  materials,  displayed  the  form  of  text, 
graphics,  and  hyperlinks.  In  or  out  of  class,  students  are  able  to  browse  and  download  the  web  pages  from  any 
location  with  an  Internet  connection  at  their  convenience. 

Some  important  issues  in  such  web  assisted  learning  environment  are:  (1)  whether  the  web  pages  motivate 
students  to  learn;  and  (2)  how  much  motivation  factors  contribute  to  learning  achievement.  The  theoretical  model  of 
motivation  in  this  study  is  based  on  Motivational  System  Theory  (MST),  proposed  by  Ford  (1992). 
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Theoretical  Background 
Motivational  systems  theory 

On  the  basis  of  Motivational  Systems  Theory  (MST),  the  concept  of  motivation  is  defined  as  the  organized 
pattern  of  an  individual’s  personal  goals,  personal  beliefs,  and  emotions”  (Ford,  1992).  It  is  implied  that  motivation 
represents  an  integrated  construct  that  provides  the  direction  a person  is  striving  for,  emotional  energy  to  support  or 
inhibit  behavior  change  toward  the  direction,  and  expectancies  a person  has  about  whether  he  or  she  is  able  to  reach 
the  destination. 

According  to  Ford  (1992),  personal  goals  have  two  properties,  content  and  direction.  A personal  goal’s 
content  represents  the  consequences  a person  is  aiming  to  achieve.  Goal  direction  guides  a person  to  produce  the 
expected  consequences.  A goal  also  provides  "the  regulatory  process  with  criteria  for  evaluating  the  effectiveness  of 
the  person’s  activity”  (Ford,  1992,  p.  73).  In  addition,  personal  goals  work  with  personal  evaluation  criteria  to 
identify  and  prioritize  goal  options  within  and  across  behavior  episodes.  Goals  are  always  within  a person  and  are 
often  constructed  from  the  context.  Because  a goal  cannot  be  imposed  on  a person  (Ford  & Lemer,  1992),  it  must  be 
adopted  as  a personal  goal  for  someone  to  perform  a desired  behavior. 

Personal  agency  beliefs  are  ’’evaluative  thoughts  involving  a comparison  between  a desired  consequence 
and  an  anticipated  consequence”  (Ford,  1992,  p.  125).  According  to  Ford,  there  are  two  types  of  personal  agency 
beliefs,  capability  beliefs  and  context  beliefs.  Capability  beliefs  are  evaluative  expectancies  about  whether  a person 
has  the  necessary  skills  to  achieve  the  goal;  context  beliefs  are  evaluative  expectancies  about  whether  the  person’s 
context  will  support  or  facilitate  the  process  of  goal  achievement.  According  to  Ford  (1992),  four  elements  are 
needed  to  ensure  optimal  responsive  environment:  (1)  the  environment  is  congruent  with  an  individual’s  agenda  of 
personal  goals;  (2)  the  environment  is  congruent  with  the  person’s  biological,  transactional,  and  cognitive 
capabilities;  (2)  the  environment  has  the  material  and  informational  resources  needed  for  goal  achievement;  (4)  the 
environment  provides  an  emotional  climate  to  support  or  facilitate  effective  functioning.  Directed  by  a personal 
goal,  capability  beliefs,  and  context  beliefs  ’’together  provide  the  person  with  the  information  needed  to  decide 
whether  to  initiate,  maintain,  amplify,  or  inhibit  some  pattern  of  goal-directed  activity”  (Ford,  1992,  p.  74).  Personal 
agency  beliefs  play  a crucial  role  in  reaching  those  goals  that  are  challenging  but  reachable. 

Agency  beliefs  can  be  measured  by  people’s  self-reports  concerning  their  confidence  in  a certain  behavior. 
When  they  have  strong  confidence  in  doing  something,  they  will  have  higher  expectations  about  whether  they  can  do 
it  and  whether  they  will  have  a supportive  context. 

In  Ford's  view  (1992),  emotions  consist  of  three  integrated  components,  an  affective  component,  a 
physiological  component,  and  a transactional  component.  Emotions  influence  behaviors  that  are  important  to  the 
individual’s  goals.  They  are  major  influences  on  initiating  and  shaping  goals  and  personal  belief  patterns.  They 
provide  energy  to  increase  ongoing  behavior  patterns  related  to  the  goal. 

A person’s  interest  in  doing  something  reflects  an  affective  component  of  a person’s  emotions.  An  aroused 
interest  toward  a specific  behavior  will  be  accompanied  with  physical  satisfaction  while  it  triggers  the  transactional 
component  of  emotions.  Therefore,  emotions  can  be  measured  through  how  much  interest  a person  has  in  doing  a 
certain  behavior. 

Based  on  Ford  (1992),  personal  goals,  personal  agency  beliefs,  and  emotions  are  interdependent,  and  their 
relationships  are  complex  and  nonlinear.  ’’Each  component  is  necessary,  but  none  are  sufficient  for  the  activation  of 
strong  motivational  patterns”  (Ford,  1992,  p.  80). 

Stated  by  Ford  (1992),  human  effective  functioning  is  represented  by  achievement  and  competence  in  a 
specific  context.  Achievement  is  defined  as  "the  attainment  of  a personally  or  socially  valued  goal  in  a particular 
context”  (Ford,  1992,  p.66).  The  concept  of  competence  is  defined  as  ’’the  attainment  of  relevant  goals  in  specified 
environments,  using  appropriate  means  and  resulting  in  positive  developmental  outcomes”  (Ford,  1992,  p.  67).  To 
ensure  effective  functioning,  a person  must  be  motivated,  have  relevant  skills  to  attain  the  goals  and  the  competence, 
and  supportive  biological  and  behavioral  capabilities  used  to  interact  with  an  environment  that  has  the  relevant 
resources  to  facilitate  goal  attainment. 

In  this  study,  the  three  components  of  motivation,  personal  goals,  agency  beliefs,  and  emotions  will  be 
tested.  The  relationships  among  the  three  components  and  the  learning  achievement  will  be  examined. 

Overview  of  MST  application  and  research 

The  MST  has  been  applied  to  several  educational  domains,  such  as  competence  development  (Ford,  1995), 
social  intelligence  development  (Ford,  1983),  adoption  of  microcomputers  in  early  childhood  education  (Ford, 

1988),  and  infant  attachment  (Ford  & Thompson,  1985). 

For  competence  development  in  special  and  remedial  education,  17  principles  were  used  for  motivating 
students  (Ford,  1995).  The  goal-directness  and  perception  of  control  have  been  studied  in  developing  social 
intelligence  (Ford,  1983).  The  motivational  factors  involved  in  decision  making  were  examined  in  adoption  of 
microcomputers  for  administrative  and  instructional  use  in  early  childhood  education.  A study  has  been  conducted 
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to  investigate  the  nature  and  implications  of  individual  differences  in  perceptions  of  personal  agency  and  infant 
attachment  (Ford  & Thompson,  1985). 

Although  the  MST  has  been  applied  in  educational  fields  by  its  proponents,  there  is  not  sufficient  evidence 
to  support  the  validity  of  the  theory  in  motivating  human  behaviors.  More  research  needs  to  be  done  in  various 
educational  contexts  to  provide  theoretical  and  empirical  evidence  about  the  validity  of  the  model. 

Overview  of  on-line  resources  application  and  research 

Since  the  World  Wide  Web  (WWW)  was  developed,  educators  have  used  such  resources  to  serve  as 
learning  tools  in  many  educational  settings.  On-line  Writing  Labs  have  been  used  to  provide  sources  and  personal 
contacts  for  college  students  to  practice  writing  skills  in  composition  classes  (Bergland,  1996).  Science  and  math 
skills  instruction  have  been  supported  by  scientific  labs  on  the  WWW  (Friedman  et  al.,  1996).  Simulation  software 
provided  on  the  WWW  has  been  developed  to  support  collaborative  learning  across  institutions  (Neilson  & Thomas, 
1996).  The  WWW  has  been  used  by  special  educators  to  discuss  current  issues  about  practices,  policies,  and 
research  in  special  education  (Kerr  & Dworet,  1996). 

Even  though  WWW  on-line  resources  have  been  used  in  many  educational  settings  to  enhance  learning, 
little  research  has  been  done  in  terms  of  effectiveness  of  these  resources  in  facilitating  learning  or  regarding  how 
motivational  factors  such  as  goals,  agency  beliefs,  and  emotions,  function  in  such  a rich  learning  environment. 

Rationale  for  employing  confirmatory  factor  analysis  and  structural  equation  modeling 

As  stated  in  the  Standards  for  Educational  and  Psychology  Tests , the  validity  of  an  instrument  refers  to  the 
appropriateness,  meaningfulness,  and  usefulness  of  the  specific  inferences  made  from  the  test  scores.  Arguments  for 
instrument  validity  are  based  on  judgmental  evidence  and  empirical  evidence  (Gable  &Wolf,  1993).  According  to 
Gable  and  Wolf  (1993),  judgmental  evidence  is  acquired  before  the  administration  of  the  instrument  to  the  target 
group  through  examining  the  adequacy  of  the  operational  definition  of  the  construct.  Empirical  evidence  is  obtained 
after  the  instrument  has  been  administered  to  the  target  group.  Evidence  is  provided  by  relationships  among  items 
within  the  instrument  as  well  as  the  relationships  to  instruments  measuring  similar  and  different  constructs. 

Three  commonly  identified  types  of  validity  include  content,  construct,  and  criterion-related.  According  to 
Cronbach  (1971),  content  validity  answers  the  question:  To  what  extent  do  the  items  on  the  instrument  adequately 
sample  from  the  intended  universe  of  content?  The  evidence  of  content  validity  is  generally  judgmental  and  is 
mostly  gathered  before  the  administration  of  the  instrument.  The  conceptual  and  the  operational  definitions  of  the 
affective  characteristics  are  the  focus  of  the  evidence  of  content  validity.  The  theoretical  basis  for  the  conceptual 
definitions  is  provided  through  a comprehensive  review  of  literature  done  by  the  instrument  developer.  It  is  also 
suggested  that  content  experts  review  the  adequacy  of  the  conceptual  definition  of  the  instrument.  Operational 
definition  is  the  design  of  items  based  on  the  content  specified  by  the  conceptual  definition.  It  is  essential  that  the 
operational  definitions  be  reviewed  by  the  same  five  content  experts.  Their  assessments  provide  evidence  that  the 
sampling  of  items  adequately  reflects  the  intended  universe  of  content  (Gable  &Wolf,  1993). 

Construct  validity  addresses  the  question:  To  what  extent  do  concepts  or  constructs  explain  covariation  in 
the  responses  to  the  items  on  the  instrument?  The  construct  validity  argument  focuses  on  response  data  variation 
among  items  to  provide  evidence  that  the  proposed  content  categories  actually  reflect  constructs  (Gable  &Wolf, 
1993).  Evidence  of  construct  validity  is  obtained  from  administering  the  instrument  to  a representative  sample  of 
respondents  for  which  the  instrument  was  designed.  Many  statistical  techniques  can  be  conducted  to  provide 
evidence  of  construct  validity.  One  of  the  analysis  techniques  is  confirmatory  factor  analysis. 

In  a confirmatory  factor  analysis,  the  researcher  postulates  a theoretical  model  that  the  data  are  expected  to 
fit.  The  model  describes  the  number  of  factors  to  be  derived  and  which  variables  are  related  to  each  factor.  The 
results  of  the  analysis  indicate  how  well  the  empirical  data  fit  the  proposed  model.  The  advantages  of  CFA  over 
Exploratory  Factor  Analysis  (EFA)  include:  (1)  yields  unique  factorial  solutions,  (2)  defines  a testable  model,  (3) 
indicates  the  extent  to  which  a hypothesized  model  fits  the  data,  (4)  specifies  data  on  the  model  parameters  to  aid  in 
improvement  of  the  model,  and  (5)  can  test  factorial  invariance  across  groups  (Marsh  & Hocevar,  1983).  For  these 
reasons,  the  study  has  used  CFA  to  examine  whether  the  data  fit  the  theoretical  motivation  model  proposed  by  Ford 
(1992) 

Structural  equation  modeling  (SEM)  analysis  provides  evidence  for  causal  relationships  among  different 
constructs  by  disconfirming  causal  paths.  It  may  be  applied  to  demonstrate  the  relational  paths  with  directions 
among  variables,  the  inter-correlations  among  independent  variables,  and  the  amount  of  contributions  independent 
variables  to  dependent  variables  (Kenny,  1979).  Although  a multiple  regression  analysis  is  commonly  used  to  find 
out  contributions  of  independent  variables  to  dependent  variables,  it  does  not  postulate  the  causal  paths  among 
independent  variables  and  dependent  variables.  Additionally,  the  results  of  a multiple  regression  analysis  are 
trustworthy  when  there  are  strong  multicollinearity  (correlations)  among  independent  variables  (Tabachnick  & 
Fidell,  1989).  Better  than  a multiple  regression,  a SEM  is  robust  to  multicollinearity  (Kenny,  1979).  In  this  study, 
we  attempted  to  find  out  the  hypothesized  causal  paths  and  to  minimize  errors  caused  by  the  strong  multicollinearity 
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among  independent  variables.  Therefore,  in  this  study,  a SEM  was  employed  to  examine  contributions  of  personal 
goals,  agency  beliefs,  and  emotions  to  students'  learning  achievement. 

Statement  of  the  problem 

This  study  addresses  both  theoretical  and  empirical  concerns  in  three  aspects.  First,  from  an  instrument 
development  perspective,  measuring  motivation  in  the  context  of  web-assisted  learning  has  not  been  done  in  the  past. 
In  addition,  although  some  instruments  to  measure  motivation  have  developed  (Ford,  1995),  no  one  has  employed 
confirmatory  factor  analysis  to  examine  the  structure  of  motivation  instruments.  This  study  endeavors  to  test  how  the 
students’  responses  regarding  their  personal  goals,  agency  beliefs,  and  emotions  will  fit  into  the  three-component 
model  proposed  by  Ford. 

Second,  from  a theoretical  aspect,  since  MST  was  proposed,  no  research  has  been  conducted  to  examine  the 
model  using  Structural  Equation  Modeling  (SEM).  This  study  is  intended  to  test  the  relationships  among,  personal 
goals,  agency  beliefs,  emotions,  and  learning  achievement  in  order  to  test  the  theoretical  model  of  MST. 

Third,  from  an  empirical  aspect,  the  verification  and  justification  of  a motivation  instrument  will  lead  other 
researchers  to  conduct  future  studies  in  regard  to  measuring  motivation  in  the  web-assisted  learning  environment. 

The  instrument  may  also  be  used  by  instructional  designers  or  instructors  to  assess  motivation  factors  in  web-assisted 
learning.  The  evidence  on  contributions  of  three  motivation  factors  on  students’  learning  achievement  will  provide 
instructional  designers  or  instructors  with  guidance  in  the  process  of  designing  effective  web-assisted  learning 
materials. 

Therefore,  the  problems  addressed  in  this  study  are  to  provide  both  necessary  theoretical  and  empirical 
evidence  for  the  people  who  are  willing  to  apply  the  MST  in  assisting  learning  through  on-line  resources. 

The  literature  review  of  this  study  has  three  sections:  theoretical  framework  of  MST,  an  overview  of 
research  on  MST  and  on  web  based  learning,  and  the  statistical  rationale  of  using  confirmatory  factor  analysis  (CFA) 
to  verify  motivation  instruments  and  using  SEM  to  examine  the  MST  model  proposed  by  Ford  (1992). 

Research  questions 

Three  research  questions  are  addressed  for  the  motivation  concept  and  Motivation  System  Theory  in  web 
assisted  learning.  The  first  question  focuses  on  the  construct  validity  of  the  motivation.  The  last  two  questions  are 
pertinent  to  test  the  theoretical  model  of  MST,  finding  out  how  three  motivation  factors  relate  to  the  learning 
achievement  and  how  much  the  factors  relate  to  each  other.  The  questions  are  stated  as  follows: 

1.  Are  the  responses  to  the  questionnaire  for  measuring  personal  goals,  agency  beliefs,  and  emotions  consistent  with  the 
Motivation  System  Theory  proposed  by  Ford? 

2.  Do  students'  personal  goals,  agency  beliefs,  and  emotions  of  using  the  web  materials  relate  to  learning  achievement? 

3.  Do  students'  personal  goals,  agency  beliefs,  and  emotions  relate  to  each  other? 

Methods  and  procedures 

An  instrument  was  designed  to  measure  three  components  of  motivation:  personal  goals,  agency  beliefs,  and 
emotions  (Appendix  A).  The  instrument  has  ten  items  with  three  parallel  responses.  The  three  responses  measure 
personal  goals,  agency  beliefs,  and  emotions,  respectively.  Personal  goals  are  directly  questioned  in  the  instruction. 
Personal  beliefs  are  indicated  by  the  confidence  students  have  in  using  the  web  materials  in  assisting  their  learning. 
Emotions  are  represented  by  how  much  interest  students  have  in  using  the  web  materials  to  study  the  course. 

Learning  achievement  is  measured  by  the  course  grade. 

To  ensure  content  validity  of  the  motivation  instrument,  five  content  experts  were  invited  to  screen 
instrument  items.  Two  of  them  have  expertise  in  instrument  development.  Another  two  experts  have  experience  in 
designing  web  pages  to  assist  students’  learning.  One  has  expertise  in  instructional  design  theory  relating  to  use  of 
on-line  resources  to  enhance  learning.  The  finalized  instrument  was  put  on-line  to  allow  students  to  fill  it  out  at  their 
convenience  (Appendix  B).  By  writing  their  full  name,  students  granted  their  permission  for  the  researcher  to  look  at 
their  course  grades. 

Three  hundred  and  fifty  six  students  have  participated  in  the  study.  These  students  have  attended 
undergraduate  fundamental  biology  classes  in  which  the  designed  web  pages  have  the  following  features:  (1)  there 
are  class  syllabi  to  announce  requirement  and  on-going  status  of  the  course;  (2)  there  are  hyperlinks  to  additional  on- 
line resources  in  order  to  enhance  understanding  of  biology  concepts;  (3)  there  are  on-line  interactive  exams  for 
students  to  practice  before  the  semester  quizzes  and  final  exams  (Appendix  C).  After  the  final  exam,  students  filled 
out  the  on-line  questionnaire  linked  right  before  the  exam  grade  page.  The  questionnaire  data  were  sent  directly  to 
the  researcher’s  e-mail  account. 

Cronbach  (1951)  alpha  reliability  values  for  three  components  of  the  motivation  were  obtained  by  SPSS  8.0 
to  analyze  the  internal  consistency  of  the  instrument.  A confirmatory  factor  analysis  with  Lisrel  VI  was  used  to 
examine  the  construct  validity  of  the  motivation  instrument  (Question  1).  The  causal  paths  among  the  personal 
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goals,  agency  beliefs,  emotions,  and  learning  achievement  were  tested  using  the  structural  equation  modeling 
analysis  (Question  2 and  Question  3). 

In  this  study,  forjudging  whether  the  data  model  is  consistent  with  the  theoretical  model,  we  used  three 
indexes,  X2  goodness  of  fit.  Tucker  Lewis  index,  and  Bender  Bonett  index.  The  closer  to  1 of  these  indexes,  the 
better  the  data  model  would  fit  the  theoretical  model  (Kenny,  1979). 

Results 

Initial  data  screening 

The  SPSS  8.0  statistical  package  was  used  to  conduct  the  screening  of  data.  For  30  items,  there  were  351 
complete  cases  out  of  365  participants.  Two  hundred  and  forty-nine  students  gave  permission  to  the  researcher  to 
look  at  their  course  grades.  First,  the  normality  of  the  data  was  checked.  The  N:  P ratio  for  the  30  items  was  11:1. 
Twenty-five  out  of  30  instrument  items  and  the  course  grades  had  good  normality.  Data  transformation  (Tabachnick 
& Fidell,  1989)  was  then  conducted  on  items  with  poor  normality.  After  the  transformation,  all  items  had  good 
normality. 

Cronbach  alpha  reliability  analysis  on  30  items  was  conducted  with  SPSS  8.0  on  personal  goals,  agency 
beliefs,  and  emotions  and  resulted  in  0.85,  0.89,  and  0.91  respectively. 

A confirmatory  factor  analysis  with  Lisrei  VI  was  initially  conducted  on  30  items,  but  the  data  model  did 
not  fit  regardless  of  specifications.  A structural  equation  modeling  with  Lisrei  VI  was  done  on  30  items  and  the 
course  grades.  The  original  data  model  did  not  converge.  These  indicated  that  the  designed  motivation  instrument 
and  the  empirical  data  did  not  match  or  correspond.  We  then  carefully  examined  all  30  items  on  the  instrument  by 
comparing  them  with  features  of  the  web  pages  in  all  four  courses.  We  found  out  that  some  features  of  the  web 
pages  in  the  four  courses  did  not  reflect  a few  items  on  the  instrument.  For  instance,  the  web  pages  of  three  courses 
did  not  promote  collaborative  work  as  indicated  on  item  2,  "collaborate  with  other  students".  We  then  selected  12 
items  for  further  analysis,  four  each  for  personal  goals,  agency  beliefs,  and  emotions  (Appendix  D).  These  12  items 
reflect  common  features  on  the  web  pages  of  all  four  courses.  1 

Reliability  of  the  motivation  instrument 

Cronbach  alpha  reliability  analysis  was  conducted  on  12  selected  items  for  personal  goals,  agency  beliefs, 
and  emotions.  It  resulted  in  0.84,  0.85,  and  0.89  accordingly  (Table  1). 

Table  I Factor  Loadings  in  Confirmatory  Factor  Analysis 


Personal  Goals 

Agency  Beliefs 

Emotions 

1.  Know  the  key  points. 

0.832 

0.848 

0.773 

3.  Better  understand  concepts. 

0.863 

0.834 

0.873 

8.  Prepare  for  exams. 

0.633 

0.760 

0.663 

10.  Get  a better  grade. 

0.695 

1.036 

0.653 

Note. 

N = 345 


All  loadings  are  significant  at  0.01  level. 

FI:  Personal  Goals,  Cronbach  Alpha  Reliability  Coefficient  = .84 
F2:  Agency  Beliefs,  Cronbach  Alpha  Reliability  Coefficient  = .85 
F3:  Emotions,  Cronbach  Alpha  Reliability  Coefficient  = .89 


Construct  validity  of  the  motivation  instrument 

The  confirmatory  factor  analysis  with  Lisrei  VI  was  conducted  on  12  items  (n  = 345).  As  shown  on  Table 
2,  the  original  model  had  poor  fit  (significance  of  X2  (66)  = 0.00,  adjusted  X2  (66)  goodness  of  fit  was  0.69).  After 
correlated  errors  were  considered  among  items,  the  model  had  good  fit.  X2  was  not  significant  (p=.  22).  The 
adjusted  X2  goodness  of  fit  index  was  0.95,  Tucker  Lewis  index  was  .99,  and  Bentler  Bonett  index  was  .99  (Table 
2).  The  factor  loadings  are  from  .63  to  1.03  and  they  were  all  significant  (Table  1) 
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Table  2 Confirmatory  Factor  Analysis  Indexes 


X2 

Significance  of  X2 

Degree  of  Freedom 

Adjusted  X Goodness  of  fit 

Null  Model 

2748.84 

0.000 

66 

0.122 

Original  Model 

381.73 

0.000 

51 

0.692 

Final  Model 

18.93 

0.22 

15 

0.952 

Note: 

Tucker  Lewis  Index  = 

.99 

Bentler  Bonett  Index  = 

.99 

Causal  paths  among  personal  goals , agency  beliefs , and  emotions 

The  structural  equation  modeling  analysis  was  then  performed  with  Lisrel  VI  on  the  12  selected  items  of  the 
motivation  instrument  and  the  course  grade  (n  = 245).  As  shown  on  Table  4,  the  original  model  did  not  fit 
(significance  of  X2  (78)  = 0.00,  adjusted  X2  (78)  goodness  of  fit  = 0.15).  After  the  paths  were  respecified  by 
deleting  the  non-significant  paths  and  the  measurement  model  was  considered  with  correlated  errors,  the  model  had 
good  fit  (significance  of  X2  (38)  = 0.06,  adjusted  X2  (38)  goodness  of  fit  = 0.923).  The  Tucker  Lewis  index  was 
0.98,  and  the  Bender  Bonett  index  was  0.97  (see  Table  4).  The  factor  loadings  are  from  .64  to  .87,  and  they  were  all 
significant  (Table  3).  As  the  path  diagram  shows  (Figure  1),  the  Beta  from  personal  goals  to  course  grade  was  0.30, 
the  Beta  from  personal  goals  to  emotions  was  0.78,  the  Beta  from  emotions  to  agency  beliefs  was  0.82.  These  three 
paths  were  significant  at  0.01  level. 

Table  3 Factor  Loadings  in  Structural  Equation  Modeling  Analysis 


Personal  Goals 

Agency  Beliefs 

Emotions 

1 . Know  the  key  points. 

0.813 

0.832 

0.802 

3.  Better  understand  concepts 

0.827 

0.844 

0.874 

8.  Prepare  for  exams. 

0.644 

0.766 

0.700 

10.  Get  a better  grade. 

0.660 

0.742 

0.665 

Note: 

All  loadings  are  significant  at  0.01  level. 


Table  4 Structural  Equation  Modeling  Indexes 


X5 

Significance 
of  X2 

Degree  of  Freedom 

Adjusted  X2  Goodness  of  fit 

Null  Model 

1992.91 

0.000 

78 

0.146 

Original  Model 

284.22 

0.000 

63 

0.724 

Final  Model 

53.02 

0.06 

38 

0.923 

Note: 

N = 245 

Tucker  Lewis  Index  = .98 
Bentler  Bonett  Index  = .97 


364 


344 


Relationships  among  personal  goals , agency  beliefs , and  emotions 

As  shown  on  Table  5,  the  correlation  coefficient  between  personal  goals  and  agency  beliefs  was  0.55 
(p=0.00).  The  correlation  coefficient  between  agency  beliefs  and  emotions  was  0.73  (p=0.00).  The  correlation 
coefficient  between  personal  goals  and  emotions  was  0.61  (p=0.00).  All  the  correlation  coefficients  were  significant 
at  0.01  level.  The  correlation  coefficients  were  correspondent  with  the  results  of  the  structural  equation  modeling 
analysis. 

Table  5 Correlation's  Among  Personal  Goals , Agency  Beliefs ; and  Emotions 


Personal  Goals 

Agency  Beliefs 

Agency  Beliefs 

0.614  ** 

Emotions 

0.546  ** 

0.729  ** 

Note: 

**  Correlation  is  significant  at  0.01  level. 


Discussion 

The  Cronbach  alpha  reliability  coefficients  indicated  that  the  motivation  instrument  with  12  selected  items 
was  moderately  reliable.  The  results  of  the  confirmatory  factor  analysis  on  the  12  selected  items  of  the  motivation 
instrument  implied  that  the  responses  to  the  12  items  were  fairly  consistent  with  the  theoretical  motivation  construct 
proposed  by  Ford.  Therefore,  from  a theoretical  perspective.  Ford's  concept  of  motivation  with  three  factors, 
personal  goals,  agency  beliefs,  and  emotions  was  supported. 

Empirically,  it  is  suggested  that  the  motivational  instrument  with  12  selected  items  may  be  used  by  other 
researchers  when  there  are  common  features  on  the  web  pages  used  in  their  studies.  It  is  not  recommended  that  the 
initial  motivation  instrument  with  items  that  were  not  selected  for  the  data  analysis  be  adopted  for  further  studies. 

The  data  through  structural  equation  modeling  analysis  supported  that  there  were  significant  and  direct 
causal  links  between  students'  personal  goals  and  their  course  grades,  between  personal  goals  and  emotions,  and 
between  emotions  and  personal  agency  beliefs.  These  findings  demonstrated  that  students'  personal  goals  were  the 
major  contributor  to  their  personal  interests,  their  confidence,  and  their  learning  achievement  in  such  a web  assisted 
learning  environment.  Additionally,  students'  interests  influenced  by  their  learning  goals  directly  impact  their 
confidence  regarding  use  of  the  web  pages  as  learning  tools. 

The  SEM  results  demonstrated  a gap  between  the  empirical  data  and  Ford's  theoretical  motivational  model. 
The  empirical  data  model  did  not  show  significant  causal  paths  between  students'  interests  and  their  course  grades 
and  between  their  agency  beliefs  and  their  course  grades.  Several  reasons  might  cause  these  findings.  It  might  be 
due  to  the  validity  of  the  course  grades  in  measuring  students'  learning  achievement.  The  course  grades  might  not 
truly  reflect  the  students'  learning  achievement  assisted  by  the  web  pages.  As  we  went  back  to  examine  the  course 
grades  provided  by  the  instructor,  we  found  that  the  course  grades  included  semester  exams  and  the  lab  performance. 
The  web  pages  used  in  four  courses  provided  little  help  in  improving  students'  lab  performance.  Therefore,  it  is 
more  likely  that  the  contributions  of  interests  and  agency  beliefs  in  web  assisted  learning  were  minimized  when  the 
lab  performance  was  considered  to  be  part  of  students'  learning  achievement.  If  we  had  used  solely  the  exam  grades 
to  measure  students'  learning  achievement  mainly  assisted  by  web  pages,  we  might  have  found  different  results.  In 
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future  studies,  it  is  recommended  that  the  validity  of  learning  achievement  should  be  carefully  examined  before  the 
data  analysis. 

Another  reason  might  be  the  reliability  of  the  motivation  instrument  with  the  12  selected  items.  As  the 
reliability  coefficients  are  around  0.85  for  N:P  ratio  of  1 1:1,  it  showed  relatively  good  reliability  of  the  motivation 
instrument  but  not  high  reliability.  The  higher  the  reliability,  the  more  accurate  the  results  would  be. 

Finally,  there  was  a lack  of  empirical  support  for  Ford’s  MST  in  such  a web  assisted  learning  environment. 

It  might  be  questionable  that  three  components,  personal  goals,  agency  beliefs,  and  emotions,  all  worked  together 
and  contributed  to  learning  achievement  (Ford,  1992).  It  might  be  true  that  only  the  component  of  personal  goals  not 
agency  beliefs  or  emotions  is  the  major  direct  contributor  to  learning  achievement. 

Based  on  the  evidence  provided  from  this  study,  we  are  confident  in  saying  that  at  least  students’  personal 
goals  in  web  assisted  learning  were  the  major  contributors  to  their  learning  achievement.  Therefore,  when  they 
design  web  pages  to  assist  students’  learning,  instructors  or  web  designers  should  mainly  focus  on  designing  the 
features  of  the  pages  that  would  help  students  to  achieve  their  specific  learning  goals.  These  learning  goals  are  stated 
as,  knowing  the  key  points,  better  understanding  concepts,  preparing  for  exams,  and  getting  better  grades. 

As  the  research  study  was  conducted  on  four  biology  courses  assisted  by  web  pages,  the  results  were  not 
generalizable  to  other  courses  assisted  by  web  pages  or  similar  biology  courses  with  different  features  from  those  in 
our  study.  To  generalize  the  research  findings  to  different  courses,  we  would  like  to  continue  this  study  with  the  12- 
item  motivation  instrument  by  recruiting  more  students  from  variety  of  courses,  in  which  the  assisted  web  pages  have 
the  common  features  of  this  study. 
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Appendix  A 


Student  Survey 

This  survey  is  designed  to  study  your  opinions  about  using  the  web  materials  to  assist  your  study.  For  each  of  the  behaviors,  three 
questions  are  asked.  Circle  the  responses  that  best  describe  your  opinions.  There  are  no  right  or  wrong  answers  to  these  questions.  Youi* 
responses  are  strictly  anonymous.  The  data  will  be  summarized  so  that  no  individual  can  be  identified. 


*■  ■*.  ■*. “ ■" ' -iV  1:  ' Please  circle  the  following  responses.  ’ ”‘V  ■ ;| 

i Goals 

Interest 

Confidence  ] 

1.  Know  the  key  points. 

2 3 4 5 

2 3 4 

5 

1 2 

3 4 5 

2.  Collaborate  with  other  students. 

1 

2 3 4 5 

1 

2 3 4 

5 

1 2 

3 4 5 

3.  Better  understand  concepts.  1 

2 

3 4 5 

1 2 

3 

4 

5 

1 2 3 4 5 

4.  Preview  content  before  class.  12  3 4 

5 

1 2 

3 

_4_ 

5 

1 2 3 4 5 

5.  Review  content  after  class.  12  3 4 

5 

1 2 

4_ 

5 

1 2 3 4 5 

6.  Extend  knowledge  beyond  class. 

_J 

2 3 4 5 

_I 

2 3 4 

5 

1 2 

3 4 5 

7.  Do  homework  or  project. 

1 

2 3 4 5 

1 

2 3 4 

5 

1 2 

3 4 5 

8.  Prepare  for  exams.  12  3 4 

_5_ 

1 2 

T 

T 

_5_ 

1 2 3 4 5 

9.  Have  a general  picture  about 

content  of  the  class.  1 

2 

3 4 5 

1 2 

3 

5 

1 2 3 4 5 

10.  Get  a better  grade.  12  3 4 

1 2 

3_ 

4 

5_ 

1 2 3 4 5 

I grant  permission  for  the  researcher  to  look  at  my  exam  grade  for  the  explicit  purpose  of  this  study. 

Print  your  name 

Sign  your  name 
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Appendix  B:  The  On-Line  Motivational  Survey 


■ffr  Op  B at*m**i-  Qr**»  Qyac^'frfrtfcw  Bifr,-- 


! r.-i' 


u 


Please  Fill  Out  The  Survey 

Before  proceeding  to  course  grade*.  plea*e  take  a minute  to  fill  out  the  following  survey. 

This  survey  is  designed  to  study  your  opiomcms  about  using  the  web  materials  to  assist  your  study.  Circle  the  responses  that  it 
best  describe  your  opionions.  There  are  no  right  or  wrong  answers  to  these  questions.  Your  responses  are  strictly  anonymous,  ^: 
The  data  will  be  summarized  so  that  no  individual  can  be  identified  ^ 


Fast  Name:  [” 


Last  Name:  [""" 


‘Providing  your  full  name  allows  the  researcher  to  look  at  your  exam  grade  from  this  course.  If  you  don't  want  the 
researcher  to  look  at  your  exam  grade,  please  ignore  this  part 


Your  Email  | 


Class  Name:  | 


I How  strongly  do  you  think  the  following  possible  GOALS  apply  to  you? 
strongly  disagree  <=  1 2 3 4 5 


t;  Know  the  key  points.  .-•*'* ; 

2,  Collaborate  with  other  students 
3;  Better  understanding  of  concepts. 

4 Preview  content  before  class. 

5. ;Revwcoiit^'a^  . .*'•*  ; 

6.  Extend  knowledge  beyond  class. 

1:  Do  homework 'or  project.  . •• 

8.  Prepare  for  exams. 

9.  Have  a general  picture  about  the  Content  of  the  class. 

10.  Get  a better  grade 


strongly  agree 


13.  How  INTERESTED  are  you  in  using  the  web  materials  to  help  you  do  die  following? 
not  interested  <=  1 2 3 4 5 => 


very  interested 


:1.  Know  the  key  point  . 

2.  Collaborate  with  other  students 

3.  Better  understanding  of  concepts. 

4.  Preview  content  before  class. 

5.  Preview  content  after  class. 

6.  Extend  knowledge  beyond  class. 

7. '  Do  homework  or  project.  ,t  , , 

8.  Prepare  for  exams 

9.  Have  a general  picture  about  the  content  of  the  class. 
10  Get  a better  grade. 


ITT  How  CONFIDENT  are  you  that  you  use  die  web  materials  to  do  the  foflowmg? 
Not  Confidant  <=  1 2 3 4 5 => 


1.  Know  the  key  point  . 

2.  Coflab  orate  with  other  students. 

3.  Better  understanding  of  concepts. 

4.  Preview  content  before  class. 

5.  Review  content  after  class' 

6 Extend  knowledge  beyond  class. 

7.  Do homeworit or project,.../ 

8.  Prepare  for  exams. 

9.  Gave  a general  picture  about  the  content  of  the  class 

10.  Get  a better  grade. 


Vary  confidant 


Please  make  sure  you  have  circled  all  the  responses. 
Click  to  pSDBfctfT-*  | the  survey  and  proceed  to  course  grades. 

Click  to  * I your  entry, 

.'a  Thanks  for  your  helpl  cl 


raariDS^irtPone  ■ ~ ;:v ~ zzl  ■ z ,, 

HKteSl  mBEXHSSSS&tt 


rr: 


OS: 


mjmmj 
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Appendix  C:  A Sample  Web  Page 


,4-  Nelsr.rtpp  [fholoqy  10?  al  lir.nnn  Fall  1890  Index  Page] 


a «* 

a 

Hom 

Mud 

fe  . 

Irw: 

Opm 

a 

Nnt 

A 

FM 

• 

JS, 

Location:  jhttp  //www.  sp  uconn.  edu/~b<1  02vc/1 02f98?harrte1 02  htmi 

Mai  | Newtgrocpt  j Weather  j Sports  | SNET  Support  | Account  U>aga  | 
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Biology  102:  Foundations  of  Biology  — Fall  1998 

Professors  Ted  Taigen,  Tom  Terry,  and  Alice  Villalobos 
The  University  of  Connecticut 

t»rt  nvMtd  To*tdv,  22, 


Course  ® Course  Announcements  and  Updates 

Resources  o 10  a m.  Lecture  Section  (Taigen  & Terry).  Last  updated  on  Tuesday,  December  22,  1998. 

o Noon  Lecture  Section  (Taigen  & Villalobos).  Last  updated  on  Tuesday,  December  22,  1998 
® Updated  Lecture/Re admg  List 

o Ted  Taigen's  Lectures  (both  10  a m and  noon  sections)  First  half  of  semester 
O Tom  Terry's  Lectures  (10  am  section).  Second  half  of  semester, 
o Alice  Villalobos'  Lectures  (noon  section).  Second  half  of  semester. 

® Study  Guides  index  page 
® Practice  exam  index  page 
® Exam  Answer  Keys 
3 Grade 

® Lecture  Syllabus  and  Course  Requirements 

Lost  your  syllabus?  Pnnt  out  a copy  from  this  file 
® Laboratory  Syllabus 

includes  lab  schedule  and  lab  policies,  how  to  contact  your  TAs,  and  more 
® Ask  a Question  or  send  a Comment 

Useful  Links  ^ About  Ted  Taigen 
® About  Tom  Terry 
® About  Alice  Villalobos 
® Web  resources  in  Biology 

Explore  different  resources  available  on  the  Web.  C'mon, 
give  it  a try  I I guarantee  you'll  find  something  to  knock  your 
socks  off 

® Web  Access  atUConn 

Still  trying  to  find  a computer  terminal  without  waiting7  Here 
are  some  places  to  try 


"What  characterizes  the  living  world  is  the  basic  unity  that  underlies  its  tremendous  diversity  The  bvme  world 
contains  bacteria  and  whales,  viruses  and  elephants,  organisms  living  in  polar  areas  at  -20°C  and  others  living  in 
h-  -t  springs  at  70°C  All  fries e creatures,  however,  exhibit  a remarkable  unity  -->f  structure  and  function  Similar 
polymers  fulfill  similar  functions  The  genetic  code  is  the  same  and  the  translating  machinery  very  nearly  so  " 

Francois  Jacob  The  Possible  and  the  Actual  1932  Pantheon  Books 


Visiting  educators:  please  visit  the  "Resources  for  Biology  Educators"  page  to  learn  about  additional  resources  for 
the  use  of  the  Web  in  teaching  biology 


This  page  has  been  accessed  I 


I times  since  Aug  21,  1998. 
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Appendix  D 


Student  Snrvey 

This  survey  is  designed  to  study  your  opinions  about  using  the  web  materials  to  assist  vour  study.  For  each  of  behaviors,  three 
questions  are  asked.  Circle  the  responses  that  best  describe  your  opinions.  There  are  no  right  or  wrong  answers  to  these  questions.  Your 
responses  are  strictly  anonymous.  The  data  will  be  summarized  so  that  no  individual  can  be  identified. 


I Mease  do  not  circle  here. 


1.  How  strongly  do  you  think  the  following  possible  goals  apply  to  you? 

12  3 4 

not  strong< - - - - 

5 

strong 

interest 

nfident 

2.  How  interested  are  you  in  using  the  web  materials  to  help  you  do  the  followings? 

12  3 4 

no  in  teres  t<  

> very 

5 

3.  How  confident  are  you  that  you  use  the  web  materials  to  do  the  following? 

12  3 4 

not  confident< 

> high  i 

5 

> very  co; 

Please  circle  the  following  responses. 

ii 

1.  Know  the  key  points. 

1 2 3 4 5 

1 2 3 4 5 

1 

2 3 4 5 

2.  Collaborate  with  other  students. 

1 2 3 4 5 

1 2 3 4 5 

1 

2 3 4 5 

*3.  Better  understand  concepts. 

1 2 3 4 5 

12  3 4 5 

1 2 3 4 5 

4.  Preview  content  before  class. 

1 2 3 4 5 

12  3 4 5 

1 2 3 4 5 

5.  Review  content  after  class. 

1 2 3 4 5 

12  3 4 5 

1 2 3 4 5 

6.  Extend  knowledge  beyond  class. 

1 2 3 4 5 

1 2 3 4 5 

1 

2 3 4 5 

Ll 

. Do  homework  or  project. 

1 2 3 4 5 

1 2 3 4 5 

1 

2 3 4 5 

*8.  Prepare  for  exams. 

1 2 3 4 5 

1 2 3 4 5 

1 

2 3 4 5 

9.  Have  a general  picture  about 

L 

content  of  the  class. 

1 2 3 4 5 

1 2 3 4 5 

1 

2 3 4 5 

Li 

1 0.  Get  a better  grade. 

1 2 3 4 5 

1 2 3 4 5 

1 

2 3 4 5 

I grant  permission  for  the  researcher  to  look  at  my  exam  grade  for  the  explicit  purpose  of  this  study. 

Print  your  name 

Sign  your  name 

* Item  1, 3,  8,  and  10  were  selected  for  confirmatory  factory  analysis  and  structural  equation  modeling  analysis. 
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Problem  Based  Learning:  A Historical  Analysis 


Alan  Januszewski 
SUNY  Potsdam 

Robert  Pearson 
Performix,  Inc. 


Background 

In  recent  years  there  has  been  much  interest  in  the  Instructional  Design  field  around  the  concept  of 
constructivism  (e.g.  Jonassen  1991,  Wilson  1996,  Driscoll  1994,  Duffy  and  Cunningham  1996  ).  Problem  based 
teaching  and  learning  strategies  are  among  the  most  frequently  used  constructivist  design  approaches  (e.g.  West, 
Farmer,  and  Wolff  1991,  Bednar,  Cunningham,  Duffy,  and  Perry  1991,  Duffy  1996).  Problem-based  teaching  and 
learning  strategies  have  been  used  successful  with  a variety  of  learners  in  a variety  of  contexts  (see  Duffy  and 
Cunningham  1996  including  business  and  industry  (Dunlap  and  Gibson  1997),  higher  education  (Harper  and  Duffy 
1997,  Duffield  and  Grabinger  1997),  and  K-12  settings  (Fosnot  1989).  Problem-based  teaching  and  learning 
strategies  are  becoming  weii  established  as  a method  and  an  area  of  study  within  the  field  of  instructional  design. 

Problem 

An  important  but  overlooked  aspect  of  the  study  and  use  of  problem-based  teaching  learning  strategies  is  its 
history  and  the  philosophical/theoretical  implications  of  those  strategies  in  a broader  history.  At  first  this  may  not 
appear  to  be  cause  for  concern.  Problem-based  learning  can  be  seen  as  simply  an  instructional  strategy.  But  there  are 
no  value  neutral  actions  in  instruction  (Yeaman  1990,  Kwan  and  Anglin  1990).  Values  and  the  philosophical  study 
of  values  have  existed  all  through  history.  Therefore  philosophical  questions  can  be  analyzed  through  studying 
history  (Koetting  1996).  Studying  history  can  help  us  to  understand  why  we  are  the  way  we  are.  It  can  also  help  us 
to  maintain  our  traditions,  help  individuals  and  members  of  the  field  to  stick  to  their  roots.  But  it  can  also  provide  us 
with  more  options  and  help  us  to  break  out  of  past  patterns  if  it  seems  important  to  do  so  (Januszewski  1996). 

Purpose 

The  purpose  of  this  paper  is  to  examine,  through  historical  analysis,  the  values  and  beliefs  underpinning  the 
early  theories  and  applications  of  what  is  now  called  problem-based  learning  and  contrast  these  with  current  theory 
and  practice.  This  study  will  focus  on  the  first  three  decades  of  the  twentieth  century  because  during  this  time  period 
the  use  of  projects  in  education  became  popular  and  because  this  was  a period  of  time  when  ideas  from  business  and 
industry  began  to  used  in  educational  contexts.  This  analysis  will,  as  Januszewski  (1996)  suggests  put  our  current 
practice  in  a more  meaningful  context  and  potentially  suggest  new  options  and  interpretations  for  the  future 
application  of  problem  based  learning  and  teaching  strategies. 

The  project 

The  literature  dealing  with  the  field  of  education  in  the  first  three  decades  of  the  twentieth  century  shows 
that  there  were  many  interpretations  of  the  use  of  the  term  “project,,  in  the  educational  context.  As  one  might  expect, 
considerable  and  critical  discussion  of  the  meaning  and  implications  of  these  conceptions  followed. 

The  following  review  of  this  discussion  was  offered  by  a graduate  student. 

The  term  project  carries  a variety  of  meaning:  to  some,  fad;  to  some,  certain  misconceptions  involving,  for 
instance,  a confusion  between  process  and  result,  as  when  a tabourette  is  itself  called  a project;  to  some  a sound, 
but  limited  method  of  education  needing  generous  supplementation  from  other  methods,  such  as  drill;  and  to 
others,  a valid,  inclusive  method,  long  in  incidental  use,  but  now  to  be  consciously  extended.  It  is  not  difficult  to 
see  in  each  a share  of  truth.  A history  of  the  movement  would  real  at  times  a faddish  aspect;  in  its  core  of  meaning 
it  is  doubtless  very  old;  it  cannot,  alone  of  all  theories  have  escaped  misconception;  and  when  conceived  as  a 
partial  method,  it  of  course  needs  to  be  supplemented  from  without  itself.  While  some  disapprove  of  the  use  of  the 
term  itself,  they  will  perhaps  admit  that  the  main  trend  seems  to  favor  both  the  term  and  the  practice  (Herring, 

1921). 

The  differences  among  the  various  conceptions  of  the  project  method  were  both  philosophical  and  applied, 
or  perhaps  put  another  way,  both  strategic  and  tactical. 
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The  historical  backdrop  to  the  “project” 

There  are  two  individuals,  E.  L.  Thorndike  and  John  Dewey,  whose  scholarship  had  a profound  influence 
on  those  that  promoted  the  use  of  projects  in  education  as  it  did  many  educational  issues.  Thorndike  and  Dewey, 
while  they  had  substantial  disagreements,  were  part  of  a new  emphasis  on  the  scientific  method  as  a mode  of  inquiry 
when  investigating  the  subject  of  teaching.  Science  and  scientific  applications  would  soon  stand  side  by  side  with 
philosophy  in  determining  the  educational  process. 

E.  L.  Thorndike  was  a prominent  psychologist  who  sought  to  create  a science  of  education  on  which  all 
teaching  could  be  based.  Thorndike  maintained  that  there  were  three  psychological  laws  of  learning:  readiness, 
exercise  and  effect.  The  formulation  of  these  laws  were  based  on  his  emphasis  on  learner  outcomes,  outcomes  which 
could  be  quantified.  Thorndike’s  other  contributions  included  work  with  student  evaluation,  the  choice,  design  and 
organization  of  instruction.  He  was  also  an  early  advocate  of  the  investigation  of  individual  differences  among 
learners  (Thorndike,  1911,  1913). 

John  Dewey  wrote  and  commented  extensively  in  all  areas  education,  science,  and  philosophy.  His  work 
was  often  cited  by  writers  on  opposite  sides  of  the  same  educational  issue,  the  project  debate  has  proven  to  be  no 
different,  this  may  be  due  to  the  fact  that  it  was  common  place  that  everyone  talked  about  Dewey  and  no  one  read 
him.  Perhaps  a more  accurate  statement  would  be  everybody  talked  about  Dewey  but  few  people  understood  him. 

The  citations  of  the  works  of  Dewey  were  bound  to  add  credence  to  almost  any  academic  position.  A 
cynical  observer  may  liken  these  frequent,  but  uninformed,  citations  to  trotting  out  artillery  and  cannonading  the 
opposing  position  in  an  attempt  to  dislodge  their  philosophical  base  of  support.  But  this  is  a compliment  to  Dewey. 
No  other  figure  in  educational  thought  held  so  much  notoriety  in  the  American  education  community.  It  can  be 
argued  that  Dewey  has  influenced  more  educators  than  any  other  individual  of  that  time  period.  A giant  in  a field 
may  attract  disciples  who  in  turn  interpret  the  original  work  into  something  that  was  not  necessarily  the  original 
intent.  Dewey’s  writing  style  has  often  left  him  in  a position  to  be  interpreted  in  a distorted  manner  (Cremin,  1961). 

It  is  particularly  important  to  be  aware  of  the  potential  for  influence  of  Thorndike  and  Dewey  on  the  use  of 
projects  in  education  because  they  had  substantially  different  perspectives  on  the  role  of  science  in  education.  Also, 
attempts  to  infuse  projects  into  education  were  done  in  the  name  of  science. 

What  A Project  Is  Not 

J.  A.  Stevenson  provides  a useful  treatment  of  the  project  in  educational  settings.  He  proposes  his  own 
definition  of  a project.  He  compares  the  definitions  and  uses  the  term  by  others  up  to  the  time  of  his  writing.  The 
result  is  a “meta-definition”.  He  comes  to  his  definition  through  the  analysis  of  common  terminology  and  includes  as 
valid,  the  use  of  the  term  by  most  of  the  popular  proponents  of  the  project  at  the  time.  To  be  sure  not  all  whose 
definition  he  reviewed  would  agree  with  the  conclusion  he  draws.  They  would,  for  the  most  part,  agree  with  his 
determination  as  to  what  a project  is  not.  We  include  a discussion  of  what  is  not  a project  because  it  will  reveal 
some  of  the  common  ground  that  is  held  by  all  proponents  of  project  use  in  education.  This  will  also  provide  a point 
from  which  we  can  embark  on  the  explanation  of  the  uses  of  the  term  project  in  education.  Stevenson’s  definition  - 
“A  project  as  a problematic  act  carried  to  completion  in  its  natural  setting”  (Stevenson,  1925).  In  so  defining  a 
project  he  is  able  to  eliminate  the  following  practices  as  not  being  projects:  problems,  questions,  topics,  drills,  tests, 
reviews,  applications,  demonstrations,  experiments,  and  practicums.  His  elimination  of  these  terms  as  a substitute 
for  projects  is  based  on  the  fact  that  they  would  all  need  some  qualifying  adjective  which  would  change  their 
meaning  from  the  way  they  were  popularly  used  in  that  day.  The  majority  of  these  other  methods  stressed 
information  over  action,  the  subjugation  of  problem  to  a principle,  and  the  inferred  emphasis  on  an  artificial  rather 
than  natural  setting. 

Early  Project  Use  in  Education 

Prior  to  1900  the  word  project  had  been  used  for  many  years  in  business  and  in  some  specialized  forms  of 
education,  most  notably  architecture,  with  a vague  meaning.  In  1908-1910  the  unmodified  word  project  was  used  by 
R.W.  Simson  and  David  Snedden  in  their  report  to  the  Massachusetts  legislature  regarding  agriculture  courses  in 
vocational  schools.  In  approximately  the  same  time  period  (1908-1910)  the  term  project  had  begun  to  be  used  in 
professional  school  programs  such  as  medicine,  engineering,  and  journalism.  Soon  after  the  term  project  was 
creeping  into  the  vocabulary  of  educators  at  all  levels.  As  the  use  of  the  word  gained  in  popularity  it  was  inevitable 
that  many  uses  and  definitions  of  the  term  would  be  developed. 
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Various  Definitions  of  the  Project  in  Education 

Some  of  the  proposed  definitions  of,  and  explanation  of,  the  use  of  the  word  project  in  education  are  as 

follows: 

J.  A.  Stevenson's  definition  - “A  project  is  a problematic  act  carried  to  completion  in  its  natural  setting”. 
Explanation:  1)  There  is  implied  in  an  act  carried  out  to  completion  a desire  for  action  rather  than  a “passive 
absorption”  of  information;  2)  the  insistence  of  a problematic  situation  demands  reasoning  over  memorization;  3) 
by  emphasizing  a problematic  aspect  there  is  a priority  of  problem  over  principle  which  infers  a discovery  rather 
than  an  application  approach;  4)  the  natural  setting  places  value  on  the  more  life-like  environment  as  opposed  to 
an  “artificial”  setting  (Stevenson,  1925). 

W W Charters  definition  - “The  project  is  considered  to  be  an  act  carried  to  completion  in  its  natural  setting  and 
involving  the  solution  of  a relatively  complex  problem.  (Charters,  1918).  Explanation  - “The  prime  essentials  of 
the  project  are,  that  it  must  involve  the  solution  of  a problem  and  that  it  must  culminate  inaction.”  Other 
advantages  of  the  project  are  that  “ - .it  gives  training  in  locating  and  solving  problems,  it  gives  training  in  the 
technique  of  action,  and  it  teaches  subject  matter  in  connection  to  life  situations.”  (Charters,  1917).  There  is  much 
agreement  between  Charters  and  Stevenson  on  this  issue  except  for  the  amount  of  complexity  in  the  problem  to  be 
solved.  Stevenson  does  not  share  Charter’s  notion  that  a project  must  solve  a complex  problem. 

C-  W.  Stone  definition  - “A  project  is  a Life  Topic  in  which  processes  and  objects  of  learning  are  largely  manual.” 
Explanation  - Stone  defines  Life  Topics  as  units  of  experience  that  are  worthy  of  value.  Stone  also  places  an 
emphasis  on  finishing  a product.  It  also  seems  clear  that  this  limits  project  use  to  performing  physical  or  manual 
work  rather  than  solving  a problem  (Stone,  1921). 

D.  Snedden  definition  - “.  . .To  be  an  educational  project  such  as  a job  (e.g.  . .wiring  a room,  growing  a half  acre 
of  potatoes,  etc.,)  must  be  of  such  a nature  as  to  offer  large  opportunity,  not  only  for  the  acquisition  of  new  skill 
and  experience  in  practical  manipulation,  but  also  for  applications  of  old  and  learning  of  new  related  knowledge.” 
(Snedden,  1916).  Explanation  - the  use  of  the  project  as  outlined  by  Snedden  is  also  limited.  It  seems  that,  like 
Stone,  Snedden  considers  projects  only  acts  that  require  manual  activities.  He  has  a strong  emphasis  as  objective 
standards  for  the  judging  of  the  final  product  of  the  project.  Project  use  for  Snedden  requires  the  building  of  new 
knowledge  upon  older,  related  knowledge. 

J.  A.  Randall  definition  - “School  Project  - A problem  the  solution  of  which  results  in  the  production  of  some 
object,  or  knowledge  of  such  value  to  the  worker  as  to  make  the  labor  involved  seem  to  him  worth  while.” 
Explanation  - Like  Charters  and  Stevenson,  Randall  sees  a project  as  solving  a problem.  There  is  an  emphasis  on 
the  production  of  some  object,  although  obtaining  knowledge  is  included  in  his  definition.  There  is  no  seen 
objective  standard  for  outside  evaluation  discussed  by  Randall,  only  that  the  “worker”  involved  believes  that  the 
activity  was  worthwhile.  The  evaluation  is  performed  by  the  individual  learner  (Randall,  1915). 

W H.  Kilpatrick  definition  - “Project  - A wholehearted  purposeful  activity  proceeding  in  a social  environment.” 
(Kilpatrick,  1918).  Explanation  - This  definition  has  three  features  that  are  thus  far  unique:  1)  that  the  action  be 
determined  by  the  individual  learner;  2)  that  the  act  need  not  necessarily  be  carried  out  to  its  completion;  3)  that 
the  act  take  place  in  a social  setting.  Kilpatrick  identifies  four  possible  types  of  project  but  admits  that  there  may 
indeed  by  the  possibility  of  overlap  among  them.  “The  first  type  represents  those  experiences  in  which  the 
dominant  purpose  is  to  do,  to  make  or  to  effect.”  “The  second  type  of  project  may  be  defined  as  one  which 
involves  purposeful  experiencing  or  appropriation  of  an  experience.”  “The  third  kind  of  project  is  one  in  which  the 
dominating  purpose  is  to  solve  a problem,  to  unravel  and  so  compose  some  intellectual  entanglement  or  difficulty” 
“The  fourth  type  includes  experiences  in  which  the  purpose  is  to  acquire  some  item  or  degree  of  knowledge  or  skill, 
or  more  generally,  experience  in  which  a person  purposes  his  own  education  a specific  point.  (Kilpatrick,  1925). 
There  are  three  aspects  of  a project  that  can  be  identified:  1)  the  presence  of  a purpose  (the  desire);  2)  the  activity 
itself  (the  doing);  and  3)  the  outcomes  of  the  activity  (the  result  of  the  doing). 

J.  F.  Hosic  Definition  - “I  understand  by  project  a complete  unit  of  experience.  The  essential  aspects  or  elements 
of  an  experience  are,  in  the  simplest  form,  a situation  and  a response  to  it.  This,  however,  will  not  describe 
adequately  what  is  meant  by  the  type  of  experience  called  complete.  Such  a unit  includes  the  following  phases: 
situation,  problem,  purpose,  plan,  criticism  of  the  plan,  execution,  judgement  of  results,  appreciation.  This  is,  of 
course,  not  a chronological  order  strictly  speaking  as  a feeling  of  appreciation  will  spring  up  in  anticipation  of  the 
outcome,  while  on  the  other  hand,  purpose  persists  and  the  plan  is  modified  to  the  very  end.  (Hosic,  1924). 
Explanation  - Hosic,  like  Kilpatrick,  places  an  emphasis  on  the  purpose  of  the  project.  He  uses  the  terms  criticism, 
judgement,  and  appreciation  to  show  that  a particular  project  need  not  be  carried  to  the  completion  of  some  object. 
In  his  definition,  as  I have  provided  it,  he  enumerates  the  phases  of  the  project.  For  Hosic,  the  purpose  of  a project 
can  be  defined  in  terms  of  a problem. 


There  are  a number  of  definitions,  uses,  and  conceptions  of  the  project  in  education.  Those  seem  to  fall  into 
two  basic  categories.  The  first  is  the  uses  of  projects  in  the  areas  of  Agriculture,  Home  Economics,  Industrial  Arts 
and  Science  which  view  projects  as  a manual  activity  which  is  aimed  at  a pre-specified  result.  In  performing  the 
activity  and  obtaining  the  result  it  is  believed  that  the  “doer”  will  acquire  additional  knowledge  and/or  training  from 
the  process.  The  second  are  definitions  which  are  very  similar  to  the  Kilpatrick/Hosic  conception  of  project,  which 
saw  project  use  as  the  basis  to  establish  the  entire  curriculum. 

The  two  positions  on  project  usage  in  education  are  part  of  broader  theories  on  the  role  of  the  school  and 
curriculum  development,  these  are  the  social-efficiency  movement  in  education  and  child-centered  education.  The 
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social-efficiency  orientation  was  demonstrated  by  the  project  conception  of  Charters  and  Snedden.  They  hoped  to 
systematically  arrange  parts  of  the  curriculum  into  projects  in  such  a way  that  it  fit  into  their  “scientific  curriculum 
building”  mold.  Kilpatrick  and  Hosic  are  considered  part  of  the  child-centered  approach  in  their  conception  and 
development  of  project  use. 


Agreements  on  Definition 

Despite  the  marked  differences  between  these  two  camps,  there  are  some  similarities.  First,  both  groups 
agree  that  when  using  the  project  style  of  teaching  the  problem  or  purpose  that  is  to  be  addressed  in  the  project 
should  be  determined  before  any  of  the  principles  or  knowledge  to  be  learned  are  introduced.  The  introduction  of 
the  principles  or  knowledge  prior  to  the  problem  changes  the  activity  to  an  application  or  practicum,  it  is  no  longer  a 
project. 

Second,  that  principles  should  be  developed  as  they  are  needed  to  complete  the  project.  Introduction  of 
principles  prior  to  the  project  is  to  eliminate  any  change  from  the  “organized  and  systematic  treatments”  of  this 
subject  matter.  This  would  defeat  areas  of  the  agreed  upon  purposes  of  project  usage  which  is  to  avoid  presenting 
the  learners  with  dictated  information  and  exercises. 

Third,  it  is  agreed  by  members  of  both  groups  that  projects  will  be  most  successful  and  will  show  the 
highest  degree  of  effectiveness  when  the  learner  chooses  to  take  ownership  of  the  project.  The  project  must  hold  the 
students’  interest  to  be  educative.  There  does  not  appear  to  be  a substitute  for  intrinsic  motivation.  Members  of  both 
groups  agree  that  such  a motivation  is  most  desirable  and  should  be  capitalized  on  “whenever  possible.” 

Fourth,  one  of  the  great  advantages  of  the  project  style  of  instruction  is  that  it  most  resembles  real  life. 
Students  will  become  more  involved  in  project  style  work  because  they  will  be  able  to  see  the  tangible  results  of  their 
work.  This  will  in  turn  encourage  more  learning  through  a greater  effort.  A learning  spiral  will  result  as  tangible 
results  build  on  one  another. 

Fifth,  both  groups  see  the  use  of  projects  as  being  an  efficient  and  effective  way  of  promoting  student 
learning.  Students  will  learn  how  to  think,  how  to  do,  and  how  to  complete  work  once  begun.  A number  of  desirable 
skills  could  be  cultivated  using  projects  in  education. 

Finally,  problem  based  learning  could  be  either  individual  or  group  based.  It  need  not  be  one  or  the  other. 
Examples 

One  of  the  authors  has  recently  worked  on  two  instructional  development  assignments  that  illustrate  how 
modem  instructional  design  strategies  have  embraced  emerging  problem  centered  and  constructivist  perspectives.  In 
each  case,  clients  actually  use  the  term  "problem-based  learning"  to  describe  the  underlying  instructional  design 
approach.  Each  deals  with  radically  different  content  and  target  audiences.  One  is  an  initiative  in  the  public  school 
system  the  other  in  business  and  industry.  Interestingly,  both  assignments  reflect  many  of  the  characteristics  of 
"projects"  as  summarized  above. 

The  New  Brunswick  Alcohol  Awareness  Project  got  its  start  in  1997  with  a large  grant  from  the  Brewers 
Association  of  Canada.  The  Brewers  Association  designates  monies  each  year  to  support  education  and  awareness 
programs  in  the  community  around  responsible  alcohol  use.  After  extensive  collaboration  with  stakeholder  groups 
the  Brewers  Association  decided  to  ear  mark  funds  for  an  alcohol  awareness  program  aimed  at  middle  school  kids. 
The  effort  will  result  in  a set  of  curriculum  materials  that  supplements  the  meager  offerings  currently  available 
around  this  topic  in  Canadian  Schools.  Another  important  project  goal  is  that  the  final  product  leverage 
communication  and  computer  technology  available  to  kids  at  home  and  in  their  classrooms.  The  finished  curriculum 
materials  will  be  piloted  in  the  province  of  New  Brunswick  and  then  made  available  to  school  boards  across  the 
country  by  late  1999.  A group  of  curriculum  specialists,  content  experts,  instructional  designers  and  multimedia 
specialists  has  been  charged  with  building  an  engaging,  effective  technology  based  set  of  curriculum  materials. 

From  the  outset , everyone  involved  felt  strongly  that  technology  should  not  be  used  to  simply  deliver 
alcohol  related  facts  and  figures  to  learners  through  their  classroom  computers.  Instead,  the  program  developers 
believed  that  learners  should  be  engaged  through  a series  of  collaborate  activities  or  "projects".  Project  descriptions 
(as  well  as  support  materials  for  teachers  would  reside  on  a web  site.  In  addition,  the  web  site  would  contain  a rich 
repository  of  material  related  to  alcohol  use  in  society.  These  materials  will  include  graphs  showing  patterns  of 
alcohol  use  across  different  groups,  audio  clips  of  people  describing  the  reasons  they  use  alcohol  or  helping 
professionals  describing  their  interactions  with  problem  drinkers.  Learners  would  be  encouraged  to  work 
collaboratively  both  within  the  classroom  and  with  learners  at  other  schools.  Projects  would  culminate  with  an 
"artifact"  of  some  kind  --  a poster  or  newspaper  article  — that  would,  in  turn,  be  posted  to  the  "gallery"  section  of  the 
web  site.  Two  other  sections  of  the  site  will  be  designed  specifically  for  teachers  and  parents.  The  teacher’s  section 
will  include  a variety  of  strategies  around  effectively  deploying  the  web  resources  in  the  classroom.  The  parents 
section  will,  among  other  things,  provide  information  about  how  to  discuss  alcohol  related  issues  with  preteens. 

The  second  example  of  "problem  based-learning"  is  a technology  based  learning  product  for  a large 
national  police  service.  The  police  service’s  training  efforts  have  slowly  been  adopting  a problem  based  focus  over 
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the  past  five  years.  This  particular  project  represented  the  first  time  the  police  service's  problem  based  approached 
would  be  realized  through  technology-based,  self-study  learning  materials.  The  finished  product  will  be  used  with 
pre  and  inservice  police  officers.  The  police  service  wanted  a learning  product  that  would  challenge  officers  to 
reflect  on  the  way  they  resolved  challenging  and  potentially  life  threatening  situations.  Learners  would  always  gain 
access  to  foundation  content  within  the  context  of  authentic  scenarios. 

The  prototype,  now  in  development,  is  constructed  around  five  video  scenarios.  Each  scenario  unfolds 
based  on  the  decisions  a learner  makes.  At  each  decision  point,  learners  are  directed  to  foundational  content  that  will 
help  them  see  how  their  real-life  actions  are  instructed  by  underlying  principles.  At  the  conclusion  of  each  scenario, 
learners  receive  feedback  on  their  decisions  through  the  eyes  of  an  expert  colleague.  This  gives  the  debriefing 
exercise  an  enhanced  sense  of  credibility.  A collaborative  element  will  also  be  developed.  Learners  will  have  the 
opportunity  to  write  an  actual  report  based  on  their  handling  of  the  scenario.  Reports  will  be  posted  to  a web  site 
where  peers  will  have  an  opportunity  to  comment. 

Conclusion 

Much  of  the  language  of  the  field  of  education  in  the  early  twentieth  century  is  consistent  with  the  language 
that  is  used  in  the  field  of  instructional  technology,  instructional  design,  and  constructivism  today.  Examples  include 
the  need  for  efficiency,  the  use  of  objectives,  systematic  design  of  the  curriculum,  the  need  to  manage  instruction, 
individualizing  instruction,  projects,  and  problem  based  learning. 

Modem  Problem  Based  Learning  evolved  out  of  early  conceptions  of  the  project  method.  Just  as  there  were 
different  conceptions  of  the  project  method  in  education  there  are  different  usage’s  of  Problem  Based  Learning  in 
instruction,  both  in  K-12  settings  and  in  business  contexts. 

Modem  day  problem  based  learning  might  best  be  viewed  on  a continuum  where  prespecified  outcomes 
determined  by  instructors/clients  to  the  creation  of  environment  where  learners  determine  their  own  expectable 
outcomes.  Both  of  these  pole  have  roots  in  the  project  method.  In  order  to  effectively  make  decisions  about  where 
on  this  continuum  a problem  based  intervention  should  lie,  one  needs  to  understand  the  philosophical  underpinnings 
of  the  project  method  at  that  point  on  the  continuum  and  one  needs  to  understand  ones  own  philosophical  position 
regarding  teaching  and  learning. 
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Secondary  Teachers’  Professional  Uses  of  Computers 


Robin  T.  Chierot  PhD. 

California  State  University , Fresno 

The  purpose  of  the  study  described  in  this  paper  was  to  investigate  teachers'  perceptions  of  the  role 
computers  play  in  their  workplace  performance.  Subjects  were  142  secondary  public  school  teacher  who 
responded  to  a 71 -item  questionnaire.  Follow-up  telephone  interviews  were  conducted  with  15  carefully 
selected  subjects  to  provide  more  depth.  Data  were  analyzed  to  provide  a multifaceted  description  of 
teachers'  current  professional  computer  uses,  their  perceptions  of  the  effects  of  the  computer  on  their  work, 
and  what  most  facilitates  or  impedes  their  computer  use.  Subjects  generally  use  the  computer  for  typical 
tasks,  such  as  creating  instructional  materials  and  performing  administrative  tasks.  Most  perceived  their 
computer  use  as  having  a positive  impact  on  their  work,  making  them  more  professional , more  creative, 
better  informed,  and  generally  better  educators.  Surprisingly,  improved  interaction  with  colleagues  did  not 
emerge  as  a particularly  important  use  of  the  computer.  Accessibility  to  e-mail  and  Internet  access  was 
moderate  or  high  for  only  33.8%  of  the  subjects  and  few  use  the  computer  to  communicate  with  colleagues, 
a use  that  might  ease  the  isolation  of  teachers  and  foster  professional  development. 


Introduction 

“As  state  after  state  has  to  re-create  schools  so  that  they  can  meet  2 1 st  century  demands,  it  has  become 
apparent  that  their  success  depends  fundamentally  on  teachers.  What  teachers  know  and  can  do  is  the  most  important 
influence  on  what  students  can  learn”  (The  National  Commission  on  Teaching  and  America's  Future,  1996,  p.  2). 
How  teachers  go  about  accomplishing  their  daily  tasks  influences  their  current  effectiveness  and  their  continuing 
improvement.  There  are  currently  concerns  regarding  the  performance  of  teachers  (The  National  Commission  on 
Teaching  & America’s  Future,  1996)  and  acknowledgment  of  the  increasing  importance  of  teaching-related  tasks  in 
addition  to  classroom  instruction  (Hargreaves,  1994).  Despite  empirical  support  that  computer  use  improves  teacher 
productivity  (Rockman,  Pershing,  & Ware,  1992)  and  increases  feelings  of  professionalism  and  effectiveness 
(Wilson,  Hamilton,  Reslow,  & Cyr,  1994),  there  is  limited  research  that  focuses  on  how  teachers  are  integrating 
computer  practices  to  accomplish  the  many  aspects  of  their  work. 

The  purpose  of  this  study  was  to  expand  the  knowledge  base  concerning  secondary  teachers'  perceptions  of 
computer  integration  into  their  work,  the  effects  of  computer  practices  on  their  work,  and  what  most  facilitates  or 
impedes  their  effective  computer  use.  This  knowledge  base  provides  a foundation  for  further  investigation  of  ways  in 
which  the  computer  might  support  teachers’  efficiency  and  effectiveness  on  the  job. 

Conceptual  Framework 

Critical  to  investigating  teachers’  professional  computer  use  is  capturing  the  essence  of  the  meaning  of 
computer  integration  in  the  context  of  teachers’  professional  activities.  This  exploratory  study  incorporated  a 
comprehensive  approach,  conceptualizing  computer  integration  as  a multidimensional  phenomenon.  It  used  a 
framework  based  on  sociotechnical  systems  theory  and  informed  by  other  human  performance  models  from  the 
organizational  development  literature  as  the  basis  for  identifying  the  individual,  organizational,  and  environmental 
factors  that  might  influence  teachers'  professional  uses  of  computers.  The  sociotechnical  systems  framework  also 
complements  and  extends  other  conceptual  frameworks  in  the  literature  on  teachers’  work  and  on  computers  for 
educational  purposes. 

Variables  Used  in  the  Study 

The  dependent  variable  in  the  study  was  teachers’  integration  of  computers  into  their  day  to  day  work.  A 
major  focus  of  the  study  was  to  identify  and  investigate  the  many  facets  of  this  variable.  Eight  a priori  dimensions 
suggested  by  results  of  research  on  computers  in  educational  and  other  organizational  contexts  were  used  to  provide 
a learner  and  more  complete  measure  of  the  dependent  variable. 

The  independent  variables  were  the  facilitators  of  and  barriers  to  computer  use.  A range  of  individual, 
organizational,  and  environmental  factors  consistent  with  sociotechnical  systems  theory  and  empirically  supported  in 
research  on  computers  in  education  were  included.  Figure  1 shows  the  independent  variables  and  the  dimensions  of 
the  dependent  variable. 
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Figure  7.  Variables  used  in  the  study 
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Review  of  Relevant  Literature 

A study  to  identify  and  investigate  the  dimensions  that  provide  a comprehensive  picture  of  teachers’ 
professional  uses  of  computers  draws  from  the  literature  in  a variety  of  areas,  including  the  occupation  of  teaching, 
the  integration  of  integration  into  both  educational  and  organizational  contexts,  and  the  analysis  and  improvement  of 
workplace  performance. 

Measuring  Computer  Integration 

Prior  educational  research  on  computer  integration,  although  primarily  emphasizing  instructional  uses  and 
curriculum  integration  provides  useful  and  relevant  information  that  serves  as  a springboard  to  this  study.  Many 
studies  have  used  a single  measure  to  operationalize  computer  integration  into  educational  settings.  Measures  among 
the  studies  vary  and  include  frequency  (Pelgrum  & Plomp,  1991;  Zammit,  1992),  continuums  or  levels  (Carrol  & 
Perin,  1994;  Marcinkiewicz,  1993-94;  Moersch,  1995),  developmental  stages  (Baker,  Gearhart,  & Herman,  1993), 
and  styles  of  use  (Evans- Andris,  1995). 

A more  comprehensive  approach  to  conceptualizing  computer  integration  is  not  without  precedent, 
however.  For  example,  to  describe  exemplary  computer  using  teachers  Hadley  and  Sheingold  (1993)  proposed  a 
variety  of  profiles,  and  Becker  (1994)  suggested  an  index  based  on  five  components. 

Dimensions  of  Teachers'  Professional  Uses  of  Computers 

Because  teachers  routinely  perform  a wide  variety  of  tasks,  identifying  the  activities  for  which  they  use  the 
computer  is  an  important  dimension.  Reynolds  (1992)  found  that  middle  school  teachers  spend  one  half  of  their  time 
on  tasks  directly  related  to  instruction.  The  other  half  of  their  workday  is  spent  on  other  tasks,  such  as  planning  and 
preparing  for  instruction  (20  percent),  evaluating  student  learning  and  instructional  effectiveness  (15  percent),  and 
administrative  responsibilities  and  professional  development  (15  percent).  Results  of  studies  (Reynolds,  1992; 
Rosenfeld,  Reynolds,  & Bukatko,  1992;  Shedd  & Bacharach,  1991)  have  categorized  teaching  responsibilities  and 
substantiated  the  notion  that  teaching  involves  a wide  variety  of  complex  tasks. 

The  notion  of  whether  teachers  consider  their  use  of  the  computer  essential  to  their  day-to-day  work  is 
another  dimension  of  computer  integration.  According  to  Carr  (1992)  one  of  the  characteristics  that  describe  a task  is 
the  value  it  adds.  A task  can  directly  add  value,  be  ancillary  to  tasks  that  add  value,  indirectly  add  value,  or  be 
wasteful.  The  degree  to  which  a task  directly  adds  value  is  related  to  its  essentiality.  Researchers  studying  computers 
in  the  schools  generally  conceptualize  integration  as  use  for  tasks  that  are  critical  to  successful  teaching  (Becker, 
1994).  Results  of  studies  often  indicate  that  use  for  such  consequential  activities  is  not  prevalent  (Hadley  & 
Sheingold,  1993;  Marcinkiewicz,  1993-94). 

The  literature  on  technology  in  organizations  other  than  schools  provides  valuable  insights  into  a variety  of 
reasons  for  using  computers  in  the  workplace  and  the  potential  impact  of  the  computer  on  humans  and  work. 
Perspectives  on  the  purposes  for  using  computers  range  from  improving  efficiency  to  expanding  human  capacity  by 
providing  expertise,  advice,  and  external  storage  of  information.  Zuboff  (1988)  envisioned  two  possible  scenarios  for 
companies  integrating  smart  machines  into  the  workplace.  In  the  first  scenario,  which  she  termed  automate , the 
intelligence  was  located  in  the  smart  machine  at  the  expense  of  the  human  capacity  for  critical  judgment.  In  the 
second  scenario,  termed  informate , smart  machines  create  the  need  for  new  skills  and  knowledge  to  exploit  the 
potential  of  these  technologies. 

Sociotechnical  Systems  Theory  and  Computer  Integration 

Sociotechnical  systems  theory  stresses  the  importance  of  both  the  requirements  of  the  job  and  the 
organizational  culture.  According  to  this  theory,  job  performance  is  a function  of  an  appropriate  combination  of  the 
technical  requirements  of  the  job  and  the  social  and  psychological  needs  of  the  people  (Pasmore,  1988;  Trist,  1993). 
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Schein  (1994)  examined  his  research  into  the  dynamics  of  organizational  culture  and  concluded  that  a sociotec hnical 
model  is  necessary  to  understand  what  really  goes  on  in  organizational  domains  and  to  arrive  at  better  concepts  and 
theoretical  insights  into  the  characteristics  that  support  the  introduction  of  information  technology.  Hart  (1995) 
examined  the  implementation  of  technological  innovations  in  education  and  asserted  that  “the  slow  integration  of 
technology  into  schools  despite  large  infusions  of  resources  into  hardware  and  software  may  in  part  have  resulted 
from  a failure  to  recognize  the  sociotechnical  nature  of  these  innovations  ” (p.  188). 

Facilitators  and  Barriers  to  Computer  Integration 

The  influence  of  social  and  psychological  needs  on  workplace  performance  is  consistent  with  research  that 
indicates  the  importance  of  collegiality  in  the  implementation  of  innovations  in  the  schools  (McLaughlin,  1993). 
Additional  factors  influencing  the  implementation  of  technology  in  school  settings  are  summarized  in  Table  1. 
Empirical  support  is  provided  by  studies  in  which  results  suggested  the  significance  of  the  factor  as  a facilitator  or 
barrier  to  computer  use. 

Table  1.  Summary  of  Facilitators  and  Barriers  Identified  in  Studies  of  Computer  Use 


Facilitator/ 

Barrier 

Empirical  Support 

Becker 

(1994) 

Hadley  & 
Sheingold  (1993) 

Hart 

(1995) 

Pelgrum  & 
Plomp  (1991) 

Marcinkiewicz 

(1993-94) 

Rockman, 
et  al.  (1992) 

Bacharach, 
et  al.  (1986) 

Time 

X 

X 

X 

X 

X 

Staff  development 

X 

X 

X 

X 

Hardware,  software 
availability 

X 

X 

X 

Onsite  Support 

X 

X 

X 

X 

X 

Administrator  Support 

X 

X 

X 

Specific  goals 

X 

Group  Values/CoIIegiality 

X 

X 

X 

X 

Confidence/ 
Perceived  expertise 

X 

X 

X 

Relevance 

X 

Technology  Education 

X 

X 

Computer  experience 

X 

X 

X 

X 

Method 


The  study  used  both  quantitative  and  qualitative  data.  Quantitative  data  were  collected  from  a survey 
administered  to  secondary  public  school  teachers.  Subjects  in  the  study  were  142  middle  and  high  school  public 
school  teachers  at  ten  schools  in  a single  district.  Random  sampling  was  not  used.  Surveys  were  distributed  to  all 
secondary  teachers  in  the  district.  The  district  recently  developed  and  is  beginning  to  implement  a technology  plan. 

Survey  Instrument 

The  survey  consisted  of  78  items  which  measured  the  eight  a priori  dimensions  of  computer  integration, 
facilitators  and  barriers,  individual  characteristics,  and  demographic  information.  Answers  were  primarily  in  the  form 
of  Likert-type  responses  or  multiple  choice,  with  a few  short-answer  questions. 

Teaching  tasks  included  were  based  on  the  literature  on  teaching,  particularly  Reynolds  (1992),  and  from 
the  questionnaire  used  by  Rockman,  et  al.  (1992).  Items  to  measure  reasons  for  using  the  computer  were  based  on 
previous  studies  (Rockman,  et  al.,  1992;  Hadley  & Sheingold,  1993;  Pelgrum  & Plomp,  1992).  Resources  and 
barriers  were  consistent  with  sociotechnical  systems  theory  as  well  as  with  prior  research  on  computer  integration  in 
education.  The  instrument  was  revised  several  times  based  on  comments  from  various  educators.  It  was  pretested 
using  six  middle-school  classroom  teachers.  A final  revision  incorporated  their  comments. 

Data  Collection 

A copy  of  the  survey  was  distributed  to  410  teachers  at  ten  schools.  Of  this  total,  142  (34%)  were  returned. 
Participation  was  voluntary  and  responses  confidential.  Because  of  unanticipated  administrator  concern,  follow-up 
reminders  were  not  sent. 

The  67  respondents  who  indicated  they  were  willing  to  participate  in  a follow-up  telephone  interview  were 
grouped  into  four  categories:  (1)  infrequent  use  and  low  expertise;  (2)  frequent  use  and  low  expertise;  (3)  infrequent 
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use  and  high  expertise;  and  (4)  frequent  use  and  high  expertise.  Four  subjects  were  interviewed  from  groups  two 
through  four  and  three  individuals  were  interviewed  from  group  one. 

Data  Analysis 

Frequency  distributions  provided  descriptive  information  on  demographics,  on  the  eight  dimensions  of 
computer  integration,  and  on  facilitators  and  barriers.  Multiple  regression  analyses  were  used  to  provide  information 
regarding  combinations  of  facilitators  and  barriers  that  predict  two  aspects  of  computer  integration,  frequency  of  use 
and  essentiality. 

Results 

Demographic  Information 

Of  the  142  individuals  who  completed  the  questionnaire,  58.5%  were  female  and  41.5%  were  male.  Ages 
ranged  from  25  to  71  years  with  an  average  age  of  43.26  years.  The  number  of  years  of  teaching  experience  ranged 
from  one  to  46  years  with  an  average  of  16.13  years.  The  average  number  of  years  of  experience  using  the  computer 
was  7.3  years  with  a range  of  from  zero  to  29  years. 

Most  subjects  had  access  to  a computer  both  at  school  and  at  home,  but  had  limited  electronic  mail  and 
Internet  access,  as  evidenced  by  the  66.2%  of  subjects  who  rated  the  availability  of  these  resources  as  none  or  low. 
When  asked  how  they  would  rate  their  overall  computer  expertise,  23.2%  rated  themselves  nonusers  or  novices, 
40.8%  said  they  had  moderate  expertise,  and  35.9%  rated  themselves  as  above  average  or  experienced. 

Dimensions  of  Computer  Integration 

Three-fourths  of  the  subjects  (75.4%)  used  the  computer  for  work  at  least  two  to  three  times  a week;  45.1% 
used  it  daily.  Subjects  who  use  the  computer  do  so  both  at  school  (79%  during  prep  time  and  84.3%  before  or  after 
class  and  at  school  (69.3%  evenings  or  weekends  and  58.5%  during  vacation  periods).  A majority  (52.9%)  of  the 
subjects  indicated  that  they  routinely  use  the  computer  to  create  instructional  materials  A majority  did  not  use  the 
computer  for  two  of  the  tasks,  interacting  with  colleagues  (52.5%)  and  analyzing  the  effectiveness  of  specific 
lessons. 

When  asked  to  rate  how  essential  computers  are  to  their  work,  48.6%  of  the  subjects  indicated  that  they 
couldn’t  imagine  doing  their  job  without  it.  At  the  other  end  of  the  spectrum,  8.5%  said  that  they  would  do  just  as 
well  without  it.  Respondents  also  rated  how  essential  they  considered  the  computer  for  specific  teaching  tasks.  Table 
2 shows  responses  from  highest  (essential)  to  lowest  (not  important)  mean  score. 

Table  2.  Essentiality  of  the  Computer  for  Specified  Tasks 


Activity 

Not  Important 
N (%) 

Somewhat  Important 
N (%) 

Important 
N (%) 

Essential 
N (%) 

Mean 

Instructional  materials  (N=127) 

8(6.3) 

19(15.0) 

47  (37.0) 

53  (41.7) 

3.142 

Administrative  tasks  (N=122) 

16(13.1) 

14(11  5) 

39  (32.0) 

53  (43.4) 

3.057 

Develop  units/lessons  (N=126) 

16(12.7) 

23(18.3) 

43  (34.1) 

44  (34.9) 

2.913 

Gather  information  (N=123) 

20(16.3) 

28  (22.8) 

44(35.8) 

31  (25.2) 

2.699 

Professional  growth  (N=  120) 

24(19.7) 

25  (20.5) 

37  (30.3) 

36  (29.5) 

2.697 

Monitor,  assess  learning  (N=125) 

26  (20.8) 

27  (21.6) 

34  (27.2) 

38  (30.4) 

2.672 

Present  lessons  (N=l  19) 

32  (26.9) 

31  (26.1) 

37  (31.1) 

19(16.0) 

2.361 

Interact  with  colleagues  (N=120) 

46  (38.3) 

26  (21.7) 

27  (22.5) 

21  (17.5) 

2.192 

Analyze  lesson  effectiveness  (N=l  16) 

59  (50.9) 

23(19.8) 

20(17.2) 

14(12.1) 

1.905 

Of  the  101  who  responded  to  the  item  which  asked  them  to  indicate  a single  activity  they  would  fight  for  if 
they  were  limited  to  only  one  computer  activity,  the  largest  number  (29)  listed  word  processing  followed  by  record- 
keeping (28)  and  grading  (23).  Interview  subjects  would  give  a high  priority  to  technology  if  they  were  in  charge  of 
allocating  financial  resources,  although  most  hesitated  to  name  it  number  one.  Those  interviewed  mentioned  other 
important  competing  needs,  for  example,  “right  now  we’d  love  to  have  air  conditioning.”  Another  commented  that 
the  school,  which  used  to  be  an  elementary  school,  was  in  need  of  a gym.  The  teacher  continued,  “I'd  rather  have 
smaller  classes  than  computers.  I’d  chuck  my  computers  for  smaller  classes.” 

Teachers  surveyed  were  asked  to  rate  the  importance  of  eight  reasons  for  using  the  computer.  Table  3 shows 
the  distribution  of  responses  from  highest  (very  important)  to  lowest  (not  important)  mean  score. 
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Table  3.  Importance  of  Reasons  for  Using  the  Computer 


Reason 

Not  important 
N (%) 

Somewhat 
important 
N (%) 

Very  important 
N (%) 

Mean 

I can  create  more  effective  materials. 

11  (7.7) 

27(19) 

104(73.2) 

2.65 

It  saves  time. 

10(7.1) 

31  (22) 

100(70.9) 

2.63 

I can  keep  better  track  of  student  performance  and  records. 

38 (27) 

35  (24.8) 

68  (48.2) 

2.21 

I can  use  the  Internet  to  access  information  & ideas. 

46  (32.6) 

39  (27.7) 

56  (39.7) 

2.07 

It  can  help  me  do  things  I don't  currently  know  how  to  do  very 
well. 

47  (33.1) 

59  (51.5) 

36  (25.4) 

1.92 

It  helps  me  find  valuable  information  on  students. 

63  (45) 

39  (27.9) 

38  (27.1) 

1.82 

I can  communicate  and  collaborate  with  others  regardless  of 
where  they  are. 

67  (47.2) 

42  (29.6) 

33(23.2) 

1.76 

I get  lots  of  ideas  and  help  from  other  teachers. 

61  (43.6) 

57  (40.7) 

22(15.7) 

1.72 

When  asked  to  rate  various  ways  in  which  using  the  computer  might  influence  their  own  work,  a majority  of 
subjects  agreed  or  agreed  strongly  that  they  were  more  productive  (76.7%),  more  professional  (72.3%),  more 
creative  (66%),  better  informed  (54.7%),  and  generally  better  educators  (60.6%)  as  a result  of  their  computer  use. 
Only  31.4%  agreed  or  agreed  strongly  that  they  have  more  time  with  students  and  21.5%  that  they  collaborate  more 
with  other  teachers. 

Subjects  who  were  interviewed  responded  positively  when  asked  for  their  general  opinion  about  computers 
and  their  work.  One  subject  did  describe  them  as  a “mixed  blessing”  and  another  responded  “I  love  them  for  what  I 
use  them  for.  I don't  care  about  techie  stuff.”  One  teacher  mentioned  a change  in  attitude  over  the  last  seven  years.  “I 
first  refused  to  use  them  because  I didn't  know  how.  The  kids  used  them  so  I had  to  learn.”  Another  mentioned  the 
importance  of  being  a role  model.  “For  the  kids  to  see  me  use  the  computer  a lot  encourages  them  to  use  it.  They 
appeal  to  kids  I would  feel  handicapped  without  one.” 

Facilitators  and  Barriers  to  Teachers  9 Computer  Integration 

Availability  of  specified  resources  was  rated  either  moderate  or  high  by  a majority  of  respondents  except  for 
three:  (1)  release  time  to  observe  examples  (17.1  %);  (2)  E-mail  and  Internet  access  (33.8  %);  and  (3)  opportunities 
for  voluntary  inservice  (43.9  %).  Hardware  and  software  were  the  most  available  resources  with  the  accessibility  of 
each  rated  at  moderate  or  high  by  73.6%  of  respondents.  The  only  barrier  a majority  of  respondents  reported  as 
having  either  moderate  or  high  importance  was  “I  have  too  many  other  responsibilities”  (56.4%).  The  importance  of 
not  enough  staff  development  opportunities  was  rated  as  moderate  or  high  by  39.1%  of  those  responding.  Table  4 
compares  how  subjects  rated  the  accessibility  and  the  value  of  each  resource.  The  table  shows  mean  ratings  for 
accessibility  and  value,  differences  between  the  means,  and  the  results  of  paired  t-tests  for  each  resource. 

Table  4.  Comparison  of  Mean  Scores  for  Value  versus  Accessibility  of  Resources 


Resource  (Accessibility) 

Mean* 

(Value) 

Difference 

t-value 

Computer  hardware 

3.088 

3.404 

.316 

- 2.70 * 

Computer  software 

3.007 

3.228 

.221 

-2.63* 

School  administrator  support 

2.985 

3.126 

.141 

-1.62 

Help  with  hardware  or  software  problems  from  other 
teachers 

2.956 

3.252 

.296 

-3.98* 

Formal  onsite  technical  assistance  (such  as  a technology 
coordinator) 

2.918 

3.328 

.410 

-4.38* 

Conversations  among  teachers  about  uses  of  computers 

2.724 

2.873 

.149 

-1.80 

Computer-related  district  or  school  inservices 

2.699 

2.949 

.250 

-2.70* 

Specified  goals  for  teacher  computer  use  in  a School 
Improvement  Plan 

2.672 

2.844 

.172 

-2.03* 

Opportunities  for  voluntary  inservice  classes 

2.466 

2.985 

.519 

-5.35* 

E-mail  and  Internet  access 

2.237 

2.771 

.534 

-4.92* 

Release  time  to  observe  good  examples  of  computer  use  by 
other  teachers 

1.731 

2.692 

2.961 

-8.97* 

* l=None  2=Low  3=Moderate  4=High 

* pc.05 
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When  asked  why  they  thought  teachers  didn't  use  computers  more,  interview  subjects  mentioned  fear,  age, 
and  lack  of  understanding  and  knowledge  most  often.  Although  few  specifically  stated  training  when  asked  this 
question,  the  importance  of  inservice  was  mentioned  by  most  subjects  at  some  time  during  the  interviews. 

Results  of  Multiple  Linear  Regression  Analyses 

Facilitators  and  barriers  included  in  the  regression  analyses  were  available  resources,  perceived  barriers, 
and  selected  individual  characteristics.  Individual  characteristics  included  were  age,  gender,  education  level,  years  of 
computer  experience,  years  of  teaching  experience,  amount  of  prep  time,  and  self-perception  of  expertise. 

For  each  regression,  free  entry  stepwise  method  ensured  that  only  significant  predictors  remained  in  the 
regression.  Dichotomous  variables  where  one  of  the  two  responses  received  more  than  90  percent  of  the  total  are  not 
included.  Tolerance  protection  of  .30  was  used  to  prevent  excess  multicol linearity.  The  mean  value  substitution  for 
any  missing  items  resulted  in  N=142  for  all  analyses. 

Six  of  the  variables  entered  the  regression  equation  to  provide  a strong  prediction  (R^  = .5247)  of  frequency 
of  computer  use  (see  Table  5).  An  individual’s  self-perception  of  expertise  is  the  most  powerful  predictor.  The 
negative  beta  for  age  suggests  that  younger  teachers  are  associated  with  more  frequent  computer  use.  The  negative 
beta  for  relevance  indicates  that  disagreement  with  the  statement  i4I  can  do  my  work  as  well  without  a computer”  is 
associated  with  more  frequent  use.  The  negative  beta  for  “no  convenient  access  to  a computer”  indicates  that  a 
higher  rating  of  the  importance  of  this  as  a barrier  is  associated  with  more  frequent  computer  use. 

Table  5 Frequency  of  Use  Regressed  on  Individual  Characteristics , Facilitators,  and  Barriers 


Variables  in  the  equation 

Beta 

T 

SigT 

Perception  of  expertise 

.5448 

7.504 

<.0001 

Availability  of  specific  goals 

.1295 

2.113 

.0364 

Age  . 

-.1732 

-2.789 

.0061 

No  access  to  a computer 

-.2275 

-3.427 

.0008 

Software  too  complicated 

.2175 

3.041 

.0028 

Relevance 

-.1675 

-2.543 

.0121 

R = .7244  R2  = .5248 

F = 24.85 

Sig  F = <.0001 

To  guard  against  the  argument  that  frequency  and  relevance  as  separate  measures  is  tautological,  an 
additional  regression  analysis  was  performed  omitting  relevance  as  a predictor.  Results  show  four  predictors  in  this 
equation.  The  availability  of  specific  computer  goals  did  not  enter  when  relevance  was  excluded.  Perception  of 
expertise  is  still  the  most  powerful  of  the  predictors.  Table  6 shows  the  results  of  that  analysis. 

Table  6 Revised  Frequency  of  Use  Regressed  on  Individual  Characteristics , Facilitators,  and  Barriers 


Variables  in  the  equation 

Beta 

T 

SigT 

Perception  of  expertise 

.5899 

8.097 

<.0001 

Age 

-.1978 

-3.120 

.0022 

No  access  to  a computer 

-.2909 

-4.465 

<.0001 

Software  too  complicated 

.2065 

2.810 

.0057 

R = .69959  R2  = .4894 

F=  32.83  Sig  F = 

<.0001 

Three  of  the  facilitators  and  barriers  provide  a strong  prediction  (R^  = .4469)  of  the  essentiality  of 
computer  use  (see  Table  7).  As  with  the  prediction  of  frequency  of  use,  an  individual’s  self-perception  of  expertise  is 
the  most  powerful  predictor. 

Table  7 Overall  Essentiality  of  Computer  Use  Regressed  on  Individual  Characteristics,  Facilitators,  and 
Barriers 


Variables  in  the  Equation 

Beta 

T 

SigT 

Perception  of  expertise 

.5215 

6.958 

<.0001 

Relevance 

-.3882 

-5.782 

<.0001 

Software  is  too  complicated 

.1933 

2.645 

.0091 

R = .676  R2  = .457 

F = 38.71 

Sig  F=  <.0001 

Once  again  to  guard  against  the  argument  that  essentiality  and  relevance  as  separate  measures  is 
tautological,  an  additional  regression  analysis  was  performed  omitting  relevance  as  a predictor.  In  this  analysis, 
perception  of  expertise  was  still  a significant  predictor,  but  gender  and  the  barrier  of  few  other  interested  teachers 
also  entered  the  equation.  Table  8 shows  the  results  of  the  analysis. 
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Table  8 Revised  Overall  Essentiality  Regressed  on  Individual  Characteristics , Facilitators ; and  Barriers 


Variables  in  the  equation 

Beta 

T 

SigT 

Perception  of  expertise 

.5868 

8.583 

<0001 

Gender 

.2139 

3.098 

.0024 

Few  interested  teachers 

.1432 

2.078 

.0395 

R = .6045  R2  = .3654 

F = 26.4904 

Sig  F = < 0001 

Discussion  and  Implications 

This  study  focused  on  teachers’  perceptions  of  their  computer  practices  in  various  aspects  of  their  work.  Its 
comprehensive  conceptualization  of  computer  integration  was  used  to  provide  a clearer  picture  of  the  complexities 
of  this  construct. 

Although  a predominant  pattern  of  dimensions  did  not  emerge  from  the  results,  the  various  dimensions 
contributed  to  a more  complete  picture  of  the  facets  of  teachers’  professional  use  of  computers.  Teachers  in  the  study 
generally  perceived  computer  use  as  having  a positive  impact  on  their  work.  A majority  felt  they  were  more 
professional,  more  creative,  better  informed,  and  generally  better  educators  as  a result  of  their  computer  use.  Results 
suggest  that  subjects  have  somewhat  conventional  vision  regarding  the  possibilities  for  using  the  computer  to  support 
their  day  to  day  work.  Creating  more  effective  materials  and  saving  time  were  rated  as  the  most  important  reasons  for 
using  the  computer. 

Surprisingly,  collegiality  and  communication  with  colleagues  did  not  emerge  as  particularly  important 
factors.  Subjects  did  not  view  “I  can  communicate  and  collaborate  with  others”  as  an  important  reason  for  using  the 
computer.  A majority  currently  used  the  computer  to  create  instructional  materials,  while  few  (1 1.3%)  used  it  to 
communicate  with  colleagues,  a use  that  might  potentially  ease  the  isolation  of  the  profession.  The  low  percent  of 
computer  use  for  communication  might  be  attributed  to  the  accessibility  to  e-mail  and  Internet  access  which  was 
moderate  or  high  for  only  33.8%  of  the  teachers  who  responded. 

Results  of  this  study  reflect  the  dynamic  nature  of  computer  integration  and  raise  further  questions 
regarding  how  changes  in  accessible  resources  will  alter  the  nature  of  teachers’  computer  integration,  their  reasons 
for  using  computers  and  their  perceptions  of  how  the  computer  influences  their  work.  This  is  particularly  evident  in 
the  area  of  using  the  ability  of  computer  to  access  the  Internet  and  to  communicate  with  others.  The  teachers  in  this 
study  had  limited  Internet  and  email  access.  As  this  access  increases  will  it  increase  collegial  communication  and 
facilitate  professional  development?  Will  this  access  expand  the  vision  of  teachers  of  the  ways  in  which  the  computer 
might  support  their  workplace  performance?  These  are  among  the  questions  that  will  provide  insights  into  ways  that 
teachers  might  more  fully  utilize  computers  to  support  their  ongoing  development  as  effective  and  professional 
educators. 
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Teachers  of  the  2 1st  century  must  know  technology.  The  number  of  colleges  of  education  that  offer 
technology  courses  is  drastically  increasing.  Preservice  and  inservice  teachers  are  hoped  to  use 
technology  in  their  classrooms.  In  order  to  provide  them  with  the  best  instruction  on  technology  use , it  is 
important  to  explore  the  technology  courses  at  colleges  of  education.  The  purpose  of  the  study  is  to 
investigate  the  one  course  that  is  often  called  " The  Technology  Course Structures  and  contents  of  the 
course  at  twenty -five  colleges  of  education  are  examined. 


Overview 

Since  1980,  we  have  been  experiencing  a rapid  change  in  our  society  due  to  the  advance  of  technology. 
Technology  is  influencing  our  education,  schools,  and  training  of  teachers  who  will  be  teaching  in  the  classrooms  of 
the  2 1st  century. 

The  business  world  demands  schools  to  prepare  the  students  for  effective  use  of  technology  in  their  future 
work  places.  The  president  and  vice  president  of  the  United  States  have  publicly  addressed  that  every  child  in  the 
nation  should  feel  comfortable  on  the  Information  Super-Highway.  Moreover,  the  increasing  number  of  computers 
at  schools  clearly  speaks  for  the  trend.  In  the  past  ten  years,  more  than  300,000  computers  have  been  added 
(Northrup  and  Little,  1996).  According  to  a report  released  by  the  National  Council  for  Accreditation  of  Teacher 
Education  (1997),  computer-to-student  ratios  have  dropped  consistently  from  1:50  in  1985  to  1:20  in  1990  to  an 
estimated  1:9  in  1997. 

All  these  facts  indicate  that  new  technology  is  affecting  our  classroom  practice  and  the  culture  of  the 
schools.  The  technological  influence  is  challenging  our  teachers  and  will  challenge  the  two  million  new  teachers 
who  will  be  hired  over  the  next  decade.  Teachers  should  feel  comfortable  with  the  new  culture  of  the  schools  and  be 
ready  to  teach  in  the  “Information  Age.”  Do  teacher  education  programs  prepare  their  students  to  face  the 
challenges?  Unfortunately,  the  answer  is  no;  most  programs  “have  a long  way  to  go”  (NCATE,  1997,  p.  1). 

To  answer  the  urgent  need,  the  NCATE  task  force  is  assisting  colleges  of  education  to  help  their  students 
take  advantage  of  technology  for  instruction:  new  understandings,  new  approaches,  new  roles,  and  new  attitudes 
(NCATE,  1997).  The  teachers  need  to  understand  the  deep  impact  of  technology  on  the  nature  of  work,  on 
communication,  and  on  the  development  of  knowledge.  They  must  use  a wide  range  of  technological  tools  and 
software  as  part  of  their  own  instruction.  They  must  help  students  in  the  use  of  technology  to  gain  information  that 
goes  beyond  textbooks  and  teachers.  In  addition,  they  should  be  fearless  in  the  use  of  technology  and  be  life-long 
learners. 

In  response  to  the  call,  colleges  of  education  across  the  nation  become  equipped  with  new  technological 
tools.  Many  programs  are  offering  technology  courses  to  help  their  students  to  take  advantage  of  these  tools  and  to 
utilize  them  for  enhanced  learning.  However,  there  has  been  little  discussion  on  what  skills  and  knowledge  students 
must  have  to  take  advantage  of  the  technological  tools  (Old  Dominion  University,  1998).  There  is  a need  to 
investigate  the  courses  offered  in  the  colleges. 

The  following  research  focused  on  the  one  course  that  is  often  called  "The  Technology  Course."  It  is  a 
computer  course  commonly  offered  at  colleges  of  education.  The  study  investigated  how  the  course  was  taught  and 
what  was  instructed  in  the  course  at  different  colleges  of  education. 


Procedures  and  Methods 

A pilot  study  was  first  conducted  involving  nine  universities:  Arizona  State  University,  Florida  State 
University,  Harvard  University,  Indiana  University,  Massachusetts  Institute  of  Technology,  Pennsylvania  State 
University,  Stanford  University,  University  of  Virginia,  and  Yale  University.  They  were  selected  because  they  were 
either  considered  to  be  prestigious  universities  or  had  good  reputation  of  offering  outstanding  educational  technology 
programs. 

First,  phone  calls  were  made  to  the  college  of  education  of  the  universities  to  find  a course  that  was 
preparing  undergraduate  education  students  on  using  technology  for  their  future  teaching  career.  During  the  same 
time,  an  Internet  search  was  conducted  to  look  for  the  course  syllabi  of  the  nine  colleges. 

The  initial  phone  interviews  and  webpage  searches  revealed  that  each  university  had  its  own  way  of 
preparing  their  students  for  the  use  of  technology  in  instruction.  Some  universities,  such  as  Stanford  University,  did 
not  have  education  programs  for  undergraduate  students.  Therefore,  the  preliminary  research  focused  on  four  of  the 
universities  that  offered  a similar  computer  course:  Arizona  State  University,  Indiana  University,  Pennsylvania  State 
University,  and  University  of  Virginia.  The  preliminary  research  examined  how  the  course  was  taught  and  what  was 
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instructed  in  the  course.  Phone  call  interviews  were  conducted  to  obtain  detailed  information  on  this  specific  course 
at  each  university,  and  the  course  webpages  were  downloaded  for  analysis. 

Based  on  the  findings  of  the  preliminary  study,  the  researcher  expanded  the  study  and  collected  data  from  a 
larger  number  of  universities.  First,  the  researcher  collected  data  at  two  professional  conventions:  Association  for 
Educational  Communications  and  Technology  (AECT)  at  St.  Louis,  Missouri  in  February  1998  and  Society  for 
Information  Technology  and  Teacher  Education  (SITE)  at  Washington,  DC  in  March  1998.  Personal  interviews 
were  conducted  in  the  conventions,  and  e-mail  communication  followed.  Course  syllabi  were  collected  via  e-mail 
and  downloaded  from  the  Internet. 

Data  was  also  collected  from  other  universities  that  were  convenient  samples.  They  either  had  course 
syllabi  on  the  Internet  or  were  accessible  by  the  researcher  and  her  graduate  students.  No  random  sampling  was 
involved  in  the  selection.  The  data  collection  involved  collecting  syllabi  via  e-mail  and  the  web,  phone  interviews, 
and  personal  interviews. 

The  study  included  a total  of  30  universities:  Appalachian  State  University,  Arizona  State  University, 
Brigham  Young  University,  California  State  University  at  San  Bernardino,  East  Central  University,  Eastern 
Michigan  University,  Idaho  University,  Indiana  University,  Mankato  State  University,  Michigan  State  University, 
Northwestern  Oklahoma  State  University,  Oklahoma  City  University,  Oklahoma  State  University,  Pennsylvania  State 
University,  San  Diego  State  University,  Southwestern  Oklahoma  State  University,  State  University  of  New  York, 
Texas  Tech  University,  University  of  Alaska  at  Anchorage,  University  of  California  at  San  Diego,  University  of 
Central  Florida,  University  of  Central  Oklahoma,  University  of  Georgia  at  Athens,  University  of  Houston,  University 
of  Iowa,  University  of  Northern  Iowa,  University  of  Oklahoma,  University  of  Southern  Indiana,  University  of 
Virginia,  and  Valley  City  State  University.  Detailed  course  descriptions  of  25  of  the  30  universities  were  available. 
They  were  employed  to  construct  the  report  in  Table  1.  URLs  of  some  on-line  course  syllabi  available  during  the 
research  were  no  longer  accessible  when  the  paper  was  written.  The  references  list  URLs  still  active  at  the  time  of 
publication. 

The  data  collection  and  analysis  focused  on  structures  and  contents  of  the  courses.  Different  course 
structures  were  recorded.  Content  topics  like  word  processing  and  spreadsheet  that  were  examined  in  the  pilot  study 
were  employed  for  the  research.  Content  topics  were  coded  and  categorized.  During  the  analysis,  other  topics  like 
video  and  traditional  media  emerged.  The  researcher  and  her  graduate  assistants  recorded  and  analyzed  the  data 
using  spreadsheet;  they  afterwards  independently  checked  the  data  to  ensure  the  accuracy. 

Results  and  Discussions 

The  technology  courses  of  the  25  universities  were  similar  overall,  but  the  details  of  the  structure  and 
content  still  varied  from  one  university  to  another. 

Structure 

The  technology  course  was  a required  course  for  students  at  most  colleges  of  education,  but  not  required  for 
students  at  some  colleges.  Some  faculty  members  thought  that  the  course  should  be  required  while  others  did  not.  A 
faculty  member  at  Appalachian  State  University  discussed  in  an  interview  that  no  technology  course  was  required  in 
her  college  but  starting  1999  there  would  be  a "requirement  for  passing  a technology  competency  test  as  one 
criterion  for  certification."  She  noted  that  it  was  a weakness  having  a competency  test  without  offering  a required 
technology  course.  A faculty  member  at  Pennsylvania  State  University  expressed  different  opinions.  He  mentioned 
that  his  college  offered  several  computer  courses  to  undergraduate  education  students  and  that  "The  Technology 
Course"  could  be  a one-credit,  two-credit,  or  three-credit  course  depending  on  the  interest  of  the  individual  student. 
He  thought  that  it  was  not  necessary  to  require  a specific  course. 

Should  "The  Technology  Course"  at  the  colleges  of  education  be  required?  It  depends  on  the  philosophy  of 
the  faculty  members  and  nature  of  the  individual  college.  According  to  NCATE,  teachers  must  have  technology 
skills  and  knowledge.  It  is  beneficial  to  offer  a required  course  so  that  the  students  can  systematically  learn  the  skills 
and  knowledge.  Nevertheless,  some  students  might  have  mastered  the  skills  and  knowledge  before  taking  the  course; 
they  would  get  bored  or  impatient  in  class.  It  is  suggested  that  a competence  test  be  administered  and  that  students 
be  given  a waiver  for  the  course  if  they  pass  the  test.  For  the  colleges  without  such  a required  course,  it  is 
recommended  that  the  colleges  at  least  conduct  a competence  test  to  ensure  quality  of  the  graduates.  Michigan  State 
University  sets  an  example.  Students  at  this  university  are  given  the  opportunity  to  choose  a course  or  a competence 
test. 

A competence  test  can  easily  evaluate  students’  computer  skills  and  knowledge.  Can  it  effectively  assess 
students'  abilities  of  integrating  technology  into  classrooms?  Is  it  beneficial  to  have  a required  course  focusing  on 
technology  integration  after  students  master  the  computer  skills?  Or  is  it  possible  to  blend  integration  into  the 
technology  course?  Faculty  member  J.  Bauer  at  the  University  of  Northern  Colorado  used  "anchored  approach" 
(Bauer,  1998)  to  blend  integration  into  his  educational  technology  course.  The  idea  is  promising.  However,  can 
students  really  learn  a lot  of  information  like  word  processing,  spreadsheet,  database,  presentation,  multimedia, 
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telecommunications,  webpage  development,  and  technology  integration  within  one  single  course?  It  might  be  useful 
for  students  to  take  a technology  integration  course  after  the  technology  course.  In  the  technology  integration  course, 
the  instructor  can  focus  on  integration  and  at  the  same  time  review  what  students  have  learned  in  the  technology 
course. 

Two  main  types  of  teaching  structures  were  found  at  the  universities.  The  similarity  of  the  two  structures 
was  that  both  structures  consisted  of  lecture  and  lab.  In  the  lecture,  students  learned  computer  concepts,  and  they 
conducted  hands-on  activities  in  the  lab.  The  difference  between  the  two  structures  resided  in  where  the  lecture  was 
conducted.  Arizona  State  University  (ASU)  and  Indiana  University  (IU)  were  examples  of  the  two  structures. 

At  ASU,  the  lecture  was  conducted  in  a big  lecture  hall,  which  could  accommodate  numerous  students.  The 
concepts  were  taught  there  to  approximately  100  students  simultaneously.  The  hands-on  activities  were  conducted  at 
a computer  lab  in  a small  group,  with  approximately  25  students.  Students  had  two  different  instructors,  one  for  the 
lecture  and  one  for  the  lab.  The  lecture  syllabus  corresponded  to  the  lab  syllabus.  The  course  instructors  used 
identical  syllabi,  assignments,  mid-term,  and  final  examinations.  They  also  conducted  weekly  meetings  to  maintain 
consistency  of  the  course. 

At  IU,  the  course  was  conducted  in  a computer  lab  where  the  instructors  taught  both  computer  concepts  and 
skills.  The  course  instructors  were  advised  to  teach  similar  content.  However,  they  neither  used  identical  syllabi  nor 
conducted  weekly  meetings.  They  were  free  to  design  and  construct  their  own  lessons. 

Both  structures  had  their  advantages  and  disadvantages.  The  structure  at  ASU  was  organized,  and  the 
course  content  was  very  consistent  throughout  different  sections.  This  structure  allowed  the  same  content  to  be 
taught  to  more  than  300  students  each  semester.  In  addition,  this  structure  encouraged  graduate  students  who  were 
more  likely  able  to  catch  up  with  updated  computer  skills  to  teach  lab  sections  while  faculty  members  provided 
expertise  in  concepts  and  theories  during  the  lectures.  The  disadvantage  of  this  structure  was  that  the  learning  with 
numerous  students  in  a big  lecture  hall  might  be  less  effective  compared  to  the  learning  with  only  25  students  in  a 
computer  lab. 

Unlike  the  structure  at  ASU,  students  at  IU  could  access  a computer  at  any  time  in  class.  They  could  also 
receive  more  attention  from  an  instructor  compared  to  students  in  a big  lecture  hall.  The  disadvantage  of  this 
structure  was  that  students  might  not  learn  the  same  information  from  the  same  course.  With  enrollment  of  about 
500  students  per  semester  at  IU  and  involvement  of  many  instructors,  one  might  wonder  what  students  learned  in  the 
“same”  course.  An  instructor  of  the  course  at  the  university  addressed  in  a phone  interview  that  the  course  content 
varied  because  each  individual  instructor  emphasized  different  concepts  and  skills. 

The  structure  at  University  of  Georgia  was  similar  to  the  one  at  IU.  An  instructor  at  University  of  Georgia 
said  in  an  interview  that  some  students  learned  video  and  webpage  development  in  the  course  while  the  others  did 
not.  The  instructors  at  both  institutions  did  not  seem  concerned  about  consistency  of  the  course. 

Content 

At  all  universities,  the  course  content  contained  concepts  and  skills.  Concepts  included  knowledge  of 
computer  technology,  such  as  basics  of  hardware  and  telecommunication.  Students  were  also  expected  to  be  able  to 
demonstrate  and  master  skills.  For  example,  they  should  be  able  to  type  a paper  using  a word  processor,  record  their 
students’  grades  using  a spreadsheet,  and  keep  the  students’  records  using  a database. 

The  table  below  (Tab.  1)  reports  topics  that  appeared  in  the  courses  at  the  25  universities.  The  percentage 
indicates  the  ratio  of  the  universities  where  the  topics  were  taught.  For  example,  telecommunication  was  taught  at  21 
out  of  the  25  universities;  hence  84%  was  listed. 

Some  topics  were  clear  and  interpretative,  such  as  word  processing  and  spreadsheet.  Other  topics  like 
telecommunication,  computer  issues,  traditional  media,  and  curriculum  integration  were  comprised  of  subtopics.  For 
example,  telecommunication  included  Internet  search,  e-mail,  and  related  topics;  computer  issues  involved  issues 
like  copyright  and  security;  traditional  media  consisted  of  lamination,  paper-based  bulletin  board,  and  transparencies; 
curriculum  integration  covered  lesson  plan  and  activities  integrating  technology.  If  one  of  the  subtopics  was  taught 
in  the  courses,  the  topic  was  marked.  Therefore,  it  was  possible  that  “telecommunication”  was  marked  while 
“Internet  search”  was  not  taught  in  the  course. 
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Table  1.  Topics  Included  in  the  Courses  and  Percentage  of  the  Universities  That  Taught  the  Topics 


Topics 


Percentage 


Telecommunications 

84% 

Multimedia 

80% 

Spreadsheet 

76% 

Webpage  development 

72% 

Word  processing 

68% 

Presentation 

60% 

Software  evaluation 

60% 

Database 

56% 

Curriculum  integration 

52% 

Computer  issues 

48% 

Hardware  and  software 

44% 

Video 

24% 

Desktop  publishing 

20% 

Traditional  media 

8% 

It  was  found  that  the  courses  were  skill-based  and  usually  started  with  basic  tool  applications.  They 
consisted  of  word  processor,  spreadsheet,  and  database.  Many  universities  used  package  software  like  ClarisWorks 
or  MSWorks  containing  these  three  applications.  Few  universities  employed  Word,  Excel,  and  FileMaker  Pro. 
Students  were  expected  to  demonstrate  efficient  use  of  these  three  applications.  There  was  a trend  of  the  course 
away  from  word  processing  because  students  had  previously  mastered  the  skill. 

Beyond  the  three  tool  applications,  students  were  instructed  in  the  use  of  multimedia  and  presentation 
software.  Many  colleges  utilized  HyperStudio  for  multimedia  and  PowerPoint  for  presentation.  Telecommunication 
like  Internet  search  and  e-mail  became  important  components,  especially  since  more  and  more  schools  progressively 
have  set  up  their  networks.  The  advance  of  technology  constantly  causes  the  change  of  the  course  content;  more  and 
more  colleges  were  including  webpage  development  into  the  courses. 

While  many  instructors  were  following  steps  of  technology  advancement  and  included  webpage 
development  and  multimedia  production  into  the  courses,  few  instructors  advocated  the  importance  of  teaching 
traditional  media,  such  as  lamination  and  transparency.  Although  in  this  study  a low  percentage  (8%)  of  the  colleges 
was  found  to  teach  traditional  media,  the  instructors’  rationale  of  teaching  them  was  valid.  They  mentioned  that 
technology  in  many  public  schools  was  very  behind  and  that  teachers  needed  to  know  how  to  use  available 
technology,  such  as  lamination  and  Apple  II  computers.  Their  statements  pointed  out  a gap  between  technology 
preparation  and  technology  use  in  real  classrooms.  The  gap  drew  professionals’  attention  to  the  importance  of  the 
linkage  or  collaboration  between  university  faculty  members  and  schoolteachers. 

University  instructors  should  help  students  to  effectively  integrate  technology  into  classrooms.  NCATE  has 
voiced  the  importance  of  technology  integration  (NCATE,  1997).  Partnerships  between  colleges  of  education  and 
local  schools  can  help  education  students  to  understand  technology  need  and  use  in  real  educational  settings.  The 
partnership  among  San  Diego  State  University,  O’Farrell  Community  School,  and  Morse  High  School  sets  a good 
example  (San  Diego  State  University,  1998).  Dialogues  between  university  faculty  members  and  schoolteachers 
(superintendents  and  administrators)  are  essential. 

Conclusions 

Technology  development  is  affecting  our  schools.  Teacher  education  programs  are  required  to  prepare 
students  to  understand  their  new  roles,  use  new  approaches,  and  have  new  attitudes  for  teaching  in  the  Information 
Age.  "The  Technology  Course”  is  strongly  suggested  to  be  offered  to  students  at  each  college  of  education.  A 
competence  test  is  recommended  to  be  available  for  students  with  good  computer  background.  Integration  of 
technology  into  instruction  is  definitely  needed  for  preparing  professional  teachers. 

Two  main  structures  are  employed  at  the  universities,  and  each  structure  has  its  own  advantages  and 
disadvantages.  It  is  suggested  that  a college  choose  a structure,  which  fits  its  individual  institution.  The  course 
content  consists  of  computer  concepts  and  skills.  The  primary  components  include  word  processing,  spreadsheet, 
database,  multimedia,  presentation,  telecommunications,  and  webpage  development.  With  NCATE’ s directions, 
partnerships  of  colleges  with  local  schools,  and  dialogue  between  university  faculty  members  and  schoolteachers, 
educators  will  provide  students  with  skills  of  using  technology  and  with  abilities  to  integrate  technology  into 
curricula  and  instruction.  Students  with  these  skills  and  abilities  will  become  qualified  teachers  who  can  prepare  the 
children  of  the  nation  to  face  the  challenges  of  the  modem  world. 
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Portraying  the  Work  of  Instructional  Designing:  An  Activity-Oriented  Analysis 


Gail  A.  Rathbun 
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Research  on  the  practice  of  instructional  designing  has  taken  a variety  of  forms,  from  the  elaboration  of 
process  models  to  retrospective  case  studies.  Seldom  has  the  research  provided  portrayals  of  design  work 
as  it  happens,  nor  has  it  integrated  in  a coherent  way  the  interaction  of  social,  cultural,  and  historical 
influences  on  the  way  the  work  unfolds.  Activity  theory,  based  on  the  work  of  Vygotsky  and  Leont'ev,  has 
provided  researchers  in  fields  as  varied  as  software  design,  teaching,  and  organizational  analysis  with  a 
way  of  studying  work  that  integrates  context  and  the  factors  mediating  action  into  a representation  of 
human  activity.  This  study  tested  the  usefulness  of  an  activity-oriented  approach  in  describing  and 
explaining  the  work  of  designing  an  international  distance  education  course  in  business  entrepreneurship 
at  Indiana  University.  Multi-faceted  views  of  the  object/outcomes  of  work  emerged,  which  led  to  both 
breakdowns  in  the  processes  of  the  work  activity  and  to  innovation  and  modification.  The  sets  of 
assumptions  embraced  by  the  communities  of  practice  to  which  participants  belonged  strongly  influenced 
what  each  individual  considered  correct  and  appropriate  behavior.  Concrete,  empuihtiic  ways  of  doing 
guided  by  an  ethic  of  care  emerged  as  the  more  influential  forces  in  gaining  knowledge  and  guiding  action 
in  this  case.  The  researcher  suggests  that  accepting  these  forces  as  the  set  of  assumptions  underlying 
design  practice  creates  new  potential  for  re-shaping  and  renewing  the  practice  of  instructional  design,  its 
research  agendas,  and  design  education. 

Activity  theory 

The  focus  of  activity  theory,  as  elaborated  by  Leont'ev  (1981)  is  on  the  objects  of  human  activity-both  as 
products  of  activity  and  as  mediators  of  it.  This  theoretical  view  emphasizes  the  transitional  and  multifaceted  nature 
of  the  objects  of  activity,  the  key  role  of  perspectives  in  understanding  human  activity,  and  an  activity’s  situatedness 
within  a cultural  and  historical  context.  An  activity-oriented  analysis  emphasizes  a view  of  process  as  primarily  a set 
of  ongoing  negotiations  and  relationships.  These  interactions,  which  are  often  evident  as  disturbances  or 
communications  "glitches,"  spur  creativity,  innovation,  effort,  and  conflict,  which,  in  turn,  move  work  activity  to 
another  "state  of  being."  Letting  disturbances  guide  data  collection  and  analysis  has  great  potential  to  show  how  new 
activity  structures  evolve,  not  solely  through  individual  action  but  through  collective  action.  Another  principle  of  an 
activity-oriented  analysis  is  the  adoption  of  a historical  perspective  in  the  interpretation  of  the  observed  disturbances 
and  underlying  contradictions.  In  this  study,  I adopted  a developmental  view  of  the  practice  of  instructional  design 
consulting  as  it  has  evolved  since  the  1930s.  By  regarding  the  community  of  practice  of  instructional  designers  as  an 
evolving  "place"  of  development,  design  paradigms  may  be  viewed  in  a new  way.  Instead  of  viewing  them  as 
oppositional  poles  (e.g.,  either  viewing  design  as  procedural  and  linear,  based  on  a "normal"  view  of  science;  or  as 
intuitive  and  purely  empirical),  they  can  be  viewed  as  factors  necessary  to  the  development  of  design  as  a work 
activity.  By  using  activity  theory  as  a framework  for  studying  design  work,  context  and  the  dynamic  interactions  that 
drive  the  design  process  forward  are  in  the  foreground  of  the  depiction  of  work  activity  and  provide  the  structure  for 
a coherent  description. 

The  Hong  Kong  Distance  Learning  Project 

The  subject  of  this  case  study  was  the  design,  development  and  implementation  of  a distance  education 
component  for  an  existing  undergraduate  business  course  in  entrepreneurship.  The  dean  of  undergraduate  studies  in 
the  business  school  had  approached  the  instructor,  who  had  taught  the  course  for  at  least  three  previous  academic 
years,  with  the  innovative  idea  of  team-teaching  the  course  with  an  instructor  at  the  City  University  of  Hong  Kong 
(CUHK).  Video  conferencing  and  Internet-based  technologies  would  be  used  to  facilitate  collaboration  among  the 
students  and  between  the  two  teachers.  Administrators  at  both  schools  were  very  keen  on  the  idea,  especially  since 
the  planning  of  the  course  coincided  with  publicity  about  the  "handover"  of  the  former  Crown  Colony  to  the  People's 
Republic  of  China.  The  goals  of  the  Hong  Kong  Distance  Learning  Project  were  to  involve  students  in  a cross- 
cultural  learning  experience,  expose  them  to  and  train  them  in  the  use  of  distance  learning  technologies,  develop 
their  capabilities  in  global  and  international  business,  and  provide  a learning  opportunity  for  IU  faculty  and  staff. 

Nearly  30  of  the  90  students  enrolled  in  the  course  volunteered  to  participate  in  the  distance  project.  The 
majority  of  the  US  volunteers  were  freshmen  and  sophomores,  while  all  of  the  Hong  Kong  students  were  the 
equivalent  of  juniors  and  seniors.  The  instructor  formed  ten  teams  of  three  US  students  each,  and  each  team  was 
matched  with  three  of  the  Hong  Kong  students.  The  six-member  teams  developed  business  plans  together  and 
presented  their  plans  via  three  video  conferences  and  jointly-constructed  web  sites.  They  used  chat,  e-mail,  and  Alta 
Vista  Forums  as  communication  and  collaboration  tools. 


391 


370 


In  June,  when  work  began  on  the  project,  eight  people  attended  the  project's  initial  meeting.  In  attendance 
were:  the  undergraduate  dean  who  had  arranged  for  the  project,  the  US  teacher  who  would  act  as  the  project 
manager,  instructional  designer,  and  teacher;  the  assistant  director  of  instructional  technology  services  in  the 
business  school;  the  business  school's  instructional  consultant  who  would  design  and  execute  an  evaluation  of  the 
project;  an  independent  instructional  technology  consultant  who  had  been  the  teacher's  graduate  student;  a 
representative  of  the  School  of  Continuing  Studies,  which  would  provide  technical  assistance  and  some  funding;  the 
business  school’s  director  of  international  programs;  and  an  instructional  technology  consultant  from  the  Teaching 
and  Learning  Technologies  Lab  (TLTL)  who  was  conducting  a campus-wide  pilot  test  of  Alta  Vista  Forums.  As  the 
project  progressed,  the  working  group  became  smaller,  including  only  the  teacher,  the  independent  consultant,  the 
evaluator,  and  the  instructional  technology  consultant.  The  IU  teacher  communicated  directly  with  the  Hong  Kong 
teacher,  whom  he  had  met  on  a trip  to  Hong  Kong  in  May. 

Method 

Between  June  and  November  I recorded  in-person  observations  of  work  meetings  among  staff  and  faculty 
on  audiotape  and  made  transcripts  afterward.  A set  of  focusing  questions  guided  my  initial  observations.  In  addition 
to  observing  work  meetings,  I observed  a rehearsal  for  the  first  video  conference,  a training  session  for  students  in 
the  use  of  a Web-based  conferencing  system,  three  video  conferences,  two  American  class  sessions,  and  a faculty  de- 
briefing of  a pilot  test  of  a Web-based  conferencing  system.  I also  observed  a meeting  concerning  contract 
negotiations  between  the  faculty  and  the  School  of  Continuing  Studies.  These  observations  provided  background 
essential  to  understanding  the  intent  and  content  of  ongoing  communications,  and  the  actions  of  participants  as  they 
planned  and  carried  out  project  work  activity. 

I conducted  individual  interviews  with  each  participant  as  soon  as  our  schedules  permitted  (within  3 to  10 
days)  after  each  observation.  In  some  cases,  for  mutual  convenience,  at  the  participant's  request,  or  because  of  the 
location  of  the  participant  (e.g.,  Hong  Kong),  the  follow-up  interview  was  conducted  via  e-mail  or  telephone. 
Whenever  possible  (including  e-mail  interviews)  I used  the  transcript,  notes,  and  artifacts  of  the  observation  to 
stimulate  recall  of  the  observed  event.  The  purpose  of  the  interview  was  to  gain  the  participant’s  perspective  of  the 
event  and  the  project,  to  check  the  accuracy  of  my  observations,  and  in  later  interviews,  to  test  my  interpretations  of 
events  against  the  participant’s  own.  Before  each  interview  I identified  what  appeared  to  be  disturbances  by  marking 
them  in  color  on  the  printed  transcript.  Three  types  of  disturbances  quickly  emerged  as  the  most  frequently 
occurring:  change  in  thinking,  surprise,  and  misunderstanding/disagreement.  I directed  the  participant  to  these 
portions  and  asked  him  or  her  to  comment.  I asked  respondents  to  make  any  additional  comments  they  wished,  and 
to  point  out  other  disturbances  they  perceived.  I generally  followed  this  procedure  in  all  follow-up  interviews. 

Document  review , particularly  of  the  printouts  of  e-mail  or  student  conferencing  sessions,  felt  like  the 
posthumous  observation  of  a "live”  activity.  The  distinction  between  a document  and  observing  an  event  was  thus 
blurred,  but  much  activity  that  might  otherwise  have  remained  invisible  was  made  visible.  When  a set  of  Internet 
exchanges  represented  a collaborative  work  activity  that  could  have  taken  place  face  to  face,  I used  the  messages  as 
transcripts  on  which  to  base  brief  Internet  interviews,  again  letting  perceived  disturbances  guide  my  questions. 

I coded  the  disturbance-producing  exchanges  into  six  talk  types:  Advising/Consulting, 

Delineating/ Assigning,  Authority/Control,  Framing,  Technical  Talk,  and  Scenarios  (Narratives).  The  next  step  was  to 
understand  the  underlying  contradictions  in  the  activity  system  that  led  to  disturbances.  I returned  to  my  analysis  of 
each  example,  first  from  the  perspective  of  the  Object,  then  from  the  perspective  of  Subject.  Using  the  activity 
system  model  developed  by  Engestrom  (1987)  (see  below)  to  guide  the  first  levels  of  analysis,  I compared  the 
examples  of  talk,  seeking  out  fundamental  similarities  that  might  help  describe  and  explain  underlying  contradictions 
in  the  activity  system.  I then  used  the  activity  system  diagram  in  writing  my  conclusions,  supported  by  appropriate 
research  in  a variety  of  fields. 


Figure  1.  Structure  of  Human  Activity,  (Engestrom,  1987) 


Tools 


Definitions  of  components  of  activity  system 

Subject:  individual  or  subgroup  whose  actions  are  being  analyzed 

Object:  problem  or  material  receiving  the  action,  and  which  is  molded  into  an  outcome 

Rules:  norms  and  conventions,  both  implicit  and  explicit  that  constrain  action 

Tools:  mediating  instruments  and  signs  used  by  subject  on  object  to  achieve  outcome 

Community:  multiple  individuals  and  sub-groups  sharing  same  object 

Division  of  labor,  horizontal  division  of  tasks  and  vertical  division  of  power  and  status 


Further  analysis  of  the  data  from  a historical  perspective  acted  as  a "cross-examination"  of  the  data  using  a 
different  framework.  I needed  a tool  that  would  integrate  changes  over  the  years  in  the  way  people  perceive  and 
arrange  design  activities  with  a generic  description  of  what  actually  happens  when  two  or  more  people  perform 
design  activities  together.  Finding  no  suitable  pre-existing  tool,  I developed  an  analytical  tool  in  the  form  of  a matrix 
"Analysis  of  the  work  activity  from  a historical  perspective."  I then  re-examined  each  talk  type  example  and  sorted  it 
into  an  appropriate  cell  in  the  table. 

Four  activity  systems  and  their  underlying  contradictions 

Using  the  intended  outcomes  of  the  work  activities  I observed,  and  working  back  toward  the  persons  or 
entities  (Subjects)  involved,  I identified  four  distinct  concurrent  sets  of  relationships. 


• the  activity  system  of  course  design/implementation 
® the  activity  system  of  evaluation 

• the  activity  system  of  consulting 

• the  activity  system  of  relationship  building 

Disturbances  occurred  in  each  activity  system.  I looked  beyond  the  disturbances  to  first  determine  the 
factors  (Subject,  Object,  Outcome,  Tools,  Rules,  Community  of  practice,  Division  of  Labor;  see  activity  system 
diagram  above  for  a definition  of  each  factor.)  generating  the  disturbance.  I then  attempted  to  pinpoint  the  underlying 
contradiction  that  might  have  caused  the  disturbance,  and  to  explain  the  source  of  the  contradiction.  I will  highlight 
in  this  article  three  fundamental  contradictions  that  emerged  in  one  or  more  of  the  activity  systems. 

The  interplay  between  technical  and  practical  human  interests 

Disturbances  in  the  activity  systems  of  course  design/implementation  and  evaluation  arose  from 
participants'  different  perspectives  of  the  Object  and  Outcome  of  work,  and  from  the  type  of  knowledge  (a  mediating 
Tool)  most  appropriate  to  achieving  the  Outcome.  From  one  perspective  the  work  Object  (the  course)  was  a process 
(e.g.,  a learning  experience).  The  primary  alternate  perspective  of  the  work  Object  was  "course  as  product"  (a 
consumable  commodity  or  well-functioning  machine).  The  primary  perspectives  of  knowledge  as  a Tool  emerged  as 
either  a set  of  facts  (e.g.,  software  features  or  theories)  or  as  a narrative  (e.g.  hypothetical  scenarios  of  use,  narratives 
of  past  experiences).  Not  knowing  or  not  agreeing  on  which  representation  of  the  Object,  Outcome,  or  Tool  was 
appropriate  at  a given  moment  caused  disturbances.  The  contradiction  behind  these  disturbance  can  be  expressed  as 
an  interplay  between  a technical  human  interest  and  a practical  human  interest,  as  described  by  Habermas  (cited  in 
Streibel,  1991)  and  adapted  in  recent  instructional  design  research  (Grundy,  cited  in  Streibel,  1991;  Li  and  Reigeluth, 
1995).  These  interests  represent  orientations  toward  the  world,  bom  of  individual  experience,  professional  training, 
membership  in  an  ethnic  group,  the  mores  of  a society,  and  so  on. 
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Table  1:  Dispositions  in  the  various  types  of  human  interest,  from  StreibeTs  adaptation  of  Grundy  ( Streibel , 
1991) 


Technical  interest 

Practical  interest 

Orientation 

Controlling  self,  other,  and 
environment  for  external  purposes 

Understanding  self,  other  and 
environment  through  interaction 

Outcomes 

Correct  behavior 

Meaningful  action 

Authority  residing  in 

Plan 

Practitioner 

Forms  of  Logic 

Instrumental  logic 

Consensual 

The  technical  human  interest,  represented  in  this  study  by  the  product  view  of  the  Outcome  and  a view  of 
knowledge  as  a set  of  facts,  implies  a way  of  knowing  that  is  based  in  empirical  study  of  phenomena  and  experience 
for  the  purpose  of  prediction  and  control.  In  the  realm  of  technical  human  interest,  theories  are  at  first  descriptive 
and  subsequently  used  to  predict.  Knowledge,  represented  as  facts,  laws,  procedures,  is  viewed  as  an  instrument  in 
gaining  control  over  objects,  processes,  and  phenomena. 

The  practical  human  interest,  represented  here  by  a process  view  of  the  work  Object  and  a narrative  view  of 
knowledge,  implies  a way  of  knowing  that  is  emergent.  In  the  realm  of  practical  human  interest,  knowledge  is 
experience  as  expressed  in  narratives  of  what  has  been  done  or  said.  This  view  of  knowledge  aligns  itself  with  the 
human  condition:  humans  have  to  find  a way  to  be  in  the  world  and  knowledge  is  negotiated  in  the  course  of  seeking 
and  establishing  meaningful  interaction  with  other  humans  in  the  world.  In  the  practical  human  interest,  theories  that 
predict  and  control  are  viewed  as  useful  insofar  as  they  contribute  to  the  practical  goal  of  coming  to  terms  with  one’s 
lived  experience. 

One  of  the  most  striking  examples  of  the  conflict  and  interplay  of  the  technical  and  practical  human 
interests  in  this  case  study  was  the  emergence  of  two  evaluation  activity  systems.  The  two  co-existed  and  came  into 
conflict  with  each  other,  shaped  by  different  perceptions  of  the  appropriate  Object  and  Outcome  of  evaluation. 
Disturbances  arose  because  of  1)  differing  views  of  the  how  the  outcomes  of  the  evaluation  study  would  be  used,  and 
because  of  2)  different  concepts  of  the  meaning  of  ’’evaluation”  in  the  context  of  teaching  and  learning. 

The  business  school’s  instructional  consultant  designed  an  evaluation  according  to  the  rules  of  an 
experimental  study,  choosing  dependent  and  independent  variables  and  using  statistical  means  in  analyzing  the  data. 
Faculty  and  staff  expended  much  effort  on  the  creation  and  administration  of  the  project  evaluation  and  on 
accommodating  the  constraints  and  the  "rules”  of  an  experimental  study  design,  particularly  the  requirement  of 
randomly  selected  control  and  treatment  groups.  The  original  course  design  called  for  students  to  volunteer  for  the 
Hong  Kong  project.  In  return  for  the  extra  work  this  might  require,  volunteers  would  not  have  to  take  the  second 
exam.  Non-volunteers  would  not  work  in  groups  as  the  Hong  Kong  students  would.  The  teacher  change  the  design  of 
the  course  in  an  attempt  to  meet  the  requirements  of  the  experimental  design,  most  significantly  by  requiring  all  of 
the  students  to  work  in  groups  (which  doubled  his  workload),  and  by  deciding  to  form  the  experimental  groups  from 
the  pool  of  volunteers.  The  design  fell  apart  when  only  20  volunteers  came  forward.  The  teacher  did  not  wish  to  deny 
10  students  the  opportunity  to  participate  in  the  project  for  the  sake  of  forming  a control  group. 

The  other  activity  system  of  evaluation,  embedded  in  course  design  and  delivery,  was  the  evaluation  activity 
implicit  in  the  notion  of  "pilot  study."  The  teacher’s  "pilot  study,"  unlike  the  point  of  view  adopted  by  the  business 
school's  consultant,  prescribed  neither  data  collection  instruments  nor  analysis  and  interpretation  methods.  It 
prescribed  only  a "frame  of  mind"  that  permitted  experimentation  and  forgave  failure.  The  Rules  guiding  action  in 
this  embedded  evaluation  activity  required  judgment,  good  will,  and  a willingness  to  negotiate  meaning  and 
compensation  (e.g.,  no  second  exam,  "easier  grading"  for  the  Hong  Kong  project  volunteers)  with  students.  In  this 
view,  evaluation  was  an  activity  oriented  toward  practical  interests,  guiding  and  guided  by  the  teacher’s  meaningful 
action,  and  employing  consensual  logic.  At  the  same  time,  meaningful,  defensible,  usable  results  were  produced  and 
informally  shared  among  the  evaluation  team. 

The  interplay  of  ethical  and  moral  value  systems : Justice , rights , or  care? 

Because  of  the  large  size  of  the  class,  only  a portion  of  it  was  involved  in  the  Hong  Kong  project.  The  IU 
teacher  was  acutely  aware  of  his  institutional  obligation  to  deliver  the  course  described  in  the  catalog  and  of  the 
scrutiny  his  experimental,  Web-based  syllabus  and  course  materials  were  likely  to  receive  from  his  colleagues.  His 
anxiety  arose  from  knowing  that  he  would  have  to  make  changes  in  order  to  maintain  fairness  among  students  in  the 
class,  to  motivate  and  enable  students,  and  to  get  them  to  trust  that  he  knew  what  he  is  doing,  though  he  admitted  he 
was  "in  over  his  head.’’  But  the  deeper  cause  of  his  anxiety  was  the  competing  set  of  rules  that  guided  the  enactment 
of  his  contract  with  students  and  the  university. 

One  set  of  rules  can  be  identified  as  "professional  rules  of  conduct"  and  the  other  set  might  be  characterized 
as  what  has  recently  been  called  an  "ethic  of  care"  or  a "feminist  ethic"  (Gilligan,  1982;  Noddings,  1984).  On  the  one 


hand,  a standard  professional  code  of  conduct  presumes  disclosure,  diligence,  and  disinterestedness  (Shapiro,  cited 
in  Moberg,  1994).  Much  effort  is  devoted  to  evening  out  inequalities. 

On  the  other  hand,  Moberg  observes  that  in  the  feminist  model  "valued  actions  can  flourish  in  a climate  of 
inequality"  (Moberg,  1994).  Within  the  feminist  framework  the  obligations  are  to  care,  enable  emotional  work,  and 
build  authentic  trust.  Moberg  further  notes  that  the  professional  code  strives  to  mitigate  or  prevent  action,  focusing 
on  the  work  itself,  while  an  ethic  of  care  focuses  on  relationships  between  and  among  work  and  workers,  and 
provides  a basis  for  acting.  The  professional  code  of  ethics  is  inflexible  and  rigid  while  an  ethic  of  care  appears  more 
fluid  and  responsive  to  emerging  situations.  In  this  case  study  an  interesting  paradox  arose.  The  syllabus  was  a public 
and  accessible  document,  emphasizing  the  organizational  obligation,  a legal  contract,  to  deliver  a product/service, 
yet  the  web  format  allowed  changes  brought  about  by  the  teacher  acting  on  his  moral  obligation  to  care  for  those  he 
had  placed  in  a difficult  and  potentially  unfair  situation. 

A related  set  of  disturbances  arose  from  accommodating  the  vantage  points  of  students  as  consumers  and 
students  as  "co-developers"  and/or  "experimental  subjects,"  and  the  vantage  points  of  teachers  as  both  providers  of  a 
service/product  and  experimenters/test  subjects.  Accommodating  the  view  of  the  Subject  as  a guinea  pig  with  the 
view  of  the  Subject  as  either  a service  provider  or  consumer  stimulated  "emotional  work"  (Hochschild,  cited  in 
Moberg,  1994).  Because  the  experimental,  uncertain  nature  of  the  innovations  being  tried  made  teachers,  staff,  and 
students  vulnerable  to  harm  and  threatened  the  successful  completion  of  the  institutional  contract  between  them, 
many  project  participants  sought  to  ameliorate  harmful  effects  through  responding  to  personal  needs,  doing 
emotional  work,  and  taking  trust-building  action.  In  this  case  study,  participants  performed  emotional  work  by 
comforting  each  other  when  performance  fell  short  of  expectations.  For  example,  the  IU  teacher  commented  on  the 
independent  consultant's  patience  and  persistence  in  trying  to  solve  technical  problems  that  arise  during  the  AVF 
training.  Other  staff  reassured  the  teacher  that  the  outcome  of  the  project  was  not  unusual  for  courses  incorporating 
intensive  technology  use  for  the  first  time.  The  teacher  protected  and  encouraged  his  "kids"  when  they  had 
difficulties  communicating  with  CUHK,  and  when  teamwork  and  goodwill  were  in  scarce  supply,  coming  to  their 
defense  when  he  thought  they  had  been  unjustly  maligned. 

Cultural  conditioning  of  the  choice  between  doing  right  or  doing  good 

In  this  case  study,  how  an  individual  perceived  his  or  her  work  was  connected  to  his  or  her  identification 
with  a culture  or  community  of  practice.  In  my  examination  of  disturbances  occurring  within  the  activity  systems  of 
consulting  and  relationship  building  I found  evidence  of  participants'  acceptance  of  some  implicit,  culturally 
embedded  rules  which  guided  their  behavior.  Culture  and  community  of  practice  were  strong  influences  when 
combined  with  personal  styles  and  personalities,  on  how  individuals  interpreted  the  object  and  outcome  of  work 
activity,  decided  which  tools  to  select,  and  what  rules  of  action  to  follow. 

A culturally -defined  ethical  divide 

For  example,  in  the  United  States,  professionals  and  business  people  are  guided  in  their  work  by  a 
traditional  code  of  ethics  (Cavanaugh,  Moberg,  & Velasquez,  1995),  which  is  taught  and  reinforced  in  schools, 
churches,  and  other  social  groupings,  such  that  the  code  becomes  an  invisible  part  of  the  fabric  of  culture.  The 
Chinese  participants  in  this  study,  as  well,  appeared  to  accept  this  code  as  the  correct  ethical  standard  to  meet. 
Moberg  lists  as  characteristics  of  this  code: 

• Relations  between  equals 

® Emphasis  on  abstract  principles 

• Moral  agent  is  detached  and  rational 

• Value  on  rights  and  justice 

• Unilateral  giving  as  virtuous  altruism 

• Partiality  is  suspect 

Two  participants  in  particular  placed  highest  value  on  the  "good  of  the  project,"  and  believed  that  if  they 
stayed  within  their  perceived  roles,  they  could  not  achieve  the  highest  level  of  good.  At  the  same  time,  they  were 
conscious  of  assuming  roles  and  authority  beyond  the  scope  of  their  publicly  acknowledged  roles  and  sought  to 
protect  themselves  and  their  clients  from  any  negative  effects  of  their  appropriation  of  tasks  and  responsibilities.  In 
this  case  study,  participants  privately  put  emotional  effort  into  understanding  their  relationships  to  each  other  and  to 
their  work,  but  at  great  sacrifice.  They  tended  to  privilege  duty  to  "the  project"  and  to  their  profession,  over  duty  to 
each  other.  Doing  what  was  "right"  or  "correct"  often  came  into  conflict  with  doing  what  was  appropriate  to  the 
situation  or  supportive  of  trusting  relationships.  As  Moberg  points  out,  writing  about  ethics  in  business  management 
(Cavanaugh,  Moberg,  & Velasquez,  1995),  the  traditional  formulation  has  long  ignored  questions  of  the  "moral 
importance  of  interpersonal  relationships  in  business." 
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When  in  Rome...? 

The  community  of  practice  provides  a set  of  shared  values,  instilled  in  its  members  through  similar  work 
experiences,  "enculturation  emanating  from  theory",  or  the  personality  characteristics  of  people  attracted  to  the  field 
(Seels  & Richey,  1994).  Lave  and  Wenger  (1991)  elaborate: 

A community  of  practice  is  a set  of  relations  among  persons,  activity,  and  world,  over  time  and  in  relation  with 
other  tangential  and  overlapping  communities  of  practice.  A community  of  practice  is  an  intrinsic  condition  for  the 
existence  of  knowledge,  not  least  because  it  provides  the  interpretive  support  necessary  for  making  sense  of  its 
heritage  (p.  98). 

I identified  the  following  communities  of  practice  in  this  case  study: 

• the  community  of  computer  technologists 

• the  community  of  experimental  researchers 

• the  community  of  college/university  teachers 

• the  community  of  "integrators,”  such  as  instructional  technology  consultants 

• the  community  of  school  administrators,  with  US  and  Chinese  sub-groups 

• the  undergraduate  business  student  community,  with  US  and  Chinese  sub-groups 

Participants  often  tried  to  use  the  language  of  a different  community  of  practice  in  order  to  build 
relationships,  learn  and  understand  viewpoints,  or  share  knowledge.  The  disturbances  caused  by  trying  to  move  in 
and  out  of  practice  communities,  each  with  its  own  set  of  rules,  models  of  behavior,  and  language  forms  raised 
difficult  questions  for  the  participants,  who  were  mindful  of  the  project's  institutional  and  faculty/staff  development 
objectives.  Disturbances  occurred  from  efforts  to  accommodate  or  work  around  the  differences  in  each  community’s 
rules  in  order  to  accomplish  the  goals  of  the  project  and  the  institutions  involved.  For  the  "cause"  of  relationship 
building,  participants  had  to  choose  between  following  the  rules  of  their  own  respective  communities  of  practice  and 
adapting  to  the  rules  of  other  communities,  depending  on  how  explicit  and  public  the  differences  were,  and  the  pain 
or  cost  of  not  adapting. 

Analysis  from  a historical  perspective 

A developmental  view  of  day-to-day  practice  allows  a researcher  to  accommodate  and  explain  the  change 
and  contradiction  that  comprise  the  lived  experience  of  design  work.  The  activity  systems  described  here  represent  a 
point  in  time  through  which  developments  prior  to  their  existence  resonate.  Prior  conceptions  of  work  are  detected  as 
"traces"  or  "layers"  within  the  current  activity  system  structure  and  potentially  give  rise  to  disturbances.  Below  I 
single  out  the  developmental  changes  I believe  most  crucial  to  understanding  the  evolution  of  design  work  as  we 
know  it  today:  changes  in  who  participates  in  design  work  and  how  it  is  led;  and  concurrent  changes  in  the 
complexity  and  organization  of  design  work. 

Changes  in  participation  in  the  design  process 

In  the  last  30  years  there  has  been  increasing  acceptance  in  many  design  fields  of  the  appropriateness  of  the 
involvement  of  non-designers  in  design  work.  Non-designers,  such  as  clients  and  users,  are  viewed  as  not  only  the 
owners  of  a problem  and  sources  of  data  input,  but  as  active  participants  in  the  design  and  implementation  of  the 
solution  (Dorsey,  Goodrum,  & Schwen,  1997;  Ehn,  1993;  Mitchell,  1990;  Schrage,  1995).  The  concept  of 
"stakeholder"  participation,  meaning  the  consultation  and  involvement  of  relevant  individuals,  units,  and 
organizations  (or  collectivities)  during  the  life  of  a project,  has  become  widely  accepted  as  a benefit  to  designing, 
though  frequently  ignored  in  practice  (Clancey,  1993;  Goodrum,  Dorsey,  & Schwen,  1993).  As  a result  of  the 
expansion  of  participation  in  designing  wherein  clients,  users,  and  stakeholders  are  required  to  play  new  and 
unfamiliar  roles,  professional  designers  (and  most  other  professionals,  for  that  matter)  have  had  to  shift  some  of  their 
attention  from  tasks  and  objectives  to  the  nature  and  quality  of  the  process  engaged  to  accomplish  them  (see,  for 
example,  Coscarelli  & Stonewater,  1979;  Davies,  1975;  Tovar,  Gagnon,  & Schmid,  1997).  Attention  to  process 
includes  close  examination  of  interpersonal,  inter-organizational,  and  intercultural  relationships  (as  in  Campbell- 
Bonar  & Olson,  1992).  As  a result,  questions  arise  as  to  l)the  definition  of  "expert"  and  "expertise",  connected  to  the 
explicit  requirement  that  the  professional  facilitate,  teach,  and  learn  with  others;  2)  the  professional’s  choice  and 
deployment  of  tools;  and  3)  the  appropriate  and  proper  criteria  or  standards  for  regulating  or  guiding  these 
relationships  (see,  for  example,  Li  & Reigeluth,  1995;  Wilson,  1995). 

Changes  in  complexity  (specialization)  of  work 

For  his  long-term  study  of  doctor-patient  interactions  in  Espoo  Finland  Engestrom  (1993)  chose  two  of  the 
dimensions  identified  by  organization  analyst  Perrow  (1984):  complexity  of  the  work  activity  and  the  degree  of 
centralization  of  control.  He  arranged  types  of  work  activity  on  a continuum  of  development,  from  low  specialization 


in  the  organization  of  work  with  centralized  control,  to  highly  specialized  work  organization  with  decentralized 
control.  Low  specialization  (low  complexity)  means  that  an  individual  who  exerts  a high  degree  of  control  over 
assistants  (e.g.,  apprentices)  accomplishes  all  of  the  processes.  An  example  of  a more  complex  organization  of  work 
is  the  modem  industrial  factory  and  bureaucratic  offices  where  the  work  requires  complex  interactions  and 
coordination;  control  is  highly  centralized  in  order  to  maximize  efficiency.  A hybrid  form  of  craft  work  and 
bureaucratized  work  is  "humanized  activity"  where  work  remains  complex  and  requires  a high  level  of  coordination, 
but  control  of  the  work  is  decentralized  and  moves  downward  in  the  hierarchy.  A fourth  category  projects  a type  of 
work  activity  where  workers  are  empowered  to  change  the  fundamental  tasks,  goals,  products,  and  values, 
contradictions  of  their  work.  It  is  a developmental  category  that  may  grow  out  of  solutions  to  difficulties  produced  by 
the  conflict  between  the  rationalized  activity  type  and  humanized  activity  type. 

Historical  types  in  the  development  of  instructional  design  as  a work  activity 

I identified  six  possible  historical  types  in  the  development  of  instructional  design  as  a work  activity  based 
on  the  complexity  of  the  work  and  how  work  is  organized  (Perrow,  1984).  The  categories  and  ranking  scheme  are 
made  only  for  clarity  in  presentation.  The  practical  reality  is  that  within  each  historical  type  are  characteristics  of 
other  types.  Here  I have  tried  to  highlight  the  defining  characteristics  of  each  historical  type.  The  order  of 
presentation  is  roughly  from  work  that  is  the  least  specialized  in  its  organization,  under  centralized  control,  to  work 
that  is  complex  in  its  organization  and  under  de-central ized  control. 

Craft  activity 

Hollywood  director  Frank  Capra's  direction  of  training  films  in  World  War  II  might  be  a prototypical 
example.  His  vision  and  his  artistry  are  what  led  the  design  of  these  highly  successful  training  interventions.  The 
researcher/experimenter  designing  audio-visual  interventions  in  this  period  relied  heavily  on  craftpersons,  but  also 
applied  his  own  "craft"  of  experimental  design.  Observation,  surveys,  and  interviews  were  used  mainly  as  evaluative 
tools  after  the  initial  design  was  complete. 

Systematic  design 

The  Dick  and  Carey  (1996)  model  of  instructional  development  is  probably  the  best  known  example  of  the 
application  of  systematic  process  management  to  instructional  design.  Systematic  design  is  characterized  by  formal 
knowledge  elicitation  and  integration  procedures,  and  emphasizes  product  efficiency  and  effectiveness  as  measures 
of  success.  The  process  is  typically  viewed  as  occurring  in  phases:  research  (analysis)  preceding  development  of  a 
product,  followed  by  implementation  and  evaluation. 

Usability  and  "user-centered”  design 

Usability  studies  and  a user-centered  design  movement  gained  momentum  in  the  1970s  and  re -focused  the 
design  process  on  the  needs,  interests,  and  capabilities  of  the  individual  who  might  use  the  product.  With  the  advent 
of  authoring  software  and  scripting  languages,  instructional  designers  often  became  software  designers,  and  the 
language  and  techniques  of  usability  engineering  became  part  of  their  practice.  Developers  invite  members  of  the 
target  user  group  to  test  a version  of  the  product  before  it  is  completed.  The  conclusions  drawn  from  early  testing  are 
then  used  to  modify  the  product.  Sugar  and  Boling  (1995)  observe  that  the  principles  and  methods  of  user-centered 
design  roughly  correspond  to  the  needs  analysis,  learner  analysis,  and  formative  evaluation  stages  of  the  Dick  and 
Carey  (1996)  model,  but  that  usability  evaluations  are  a novel  addition  to  instructional  design  processes. 

Rapid  prototyping  (RP) 

RP  is  another  technique  borrowed  from  software  development.  Researchers  in  the  field  of  instructional 
design  (Tripp  & Bichelmeyer,  1991;  Dorsey,  Goodrum,  & Schwen,  1997)  have  explored  rapid  prototyping  as  the 
foundation  of  an  alternate  approach  to  instructional  design.  The  making  of  a product  serves  as  a way  to  both  define 
and  solve  the  problem.  Research  on  needs  and  objectives  proceeds  in  parallel  with  the  development  of  the  product.  A 
usable  product  may  thus  be  available  much  earlier  in  the  process  and  be  used  to  explore  implementation  questions. 
The  evolving  prototype  provides  a focus  for  designer-client  communication  earlier  and  more  frequently  in  the  design 
process. 

Socio-technical  design 

The  instructional  design  process  is  viewed  as  an  activity  situated  in  a changing  environment  which  includes 
people,  technology,  and  constantly  evolving  work-practice s(Goodrum  & Schwen,  1994).  Clancey  (1993)  points  out 
that  the  socio-technical  approach  includes  views  commonly  associated  with  the  systematic,  but  goes  beyond  them. 
Highly  collaborative  instructional  design  projects  such  as  those  reported  by  Cennamo,  Abell,  Chung,  Campbell,  & 
Hugg  (1995),  and  Grisdale  and  Campbell-Bonar  (1991),  are  examples  of  attempts  to  put  the  socio-technical  view 
into  practice.  In  both,  teachers  (the  content  experts)  and  designers  simultaneously  focused  on  the  major  tasks  of  the 


design  project,  and  all  were  actively  involved  throughout  the  life  of  the  project.  There  was  a conscious  effort  on  the 
part  of  designers  and  clients  alike  to  enter  each  others  community  of  practice. 

Emancipatory  design 

An  emancipatory  instructional  design  process  is  focused  on  "the  uncovering  of  factors  that  prohibit  and 
prevent  the  autonomous  actions  of  learning,  instruction,  and  design  respectively  (Li  & Reigeluth,  1995).”  The 
designer’s  role  is  to  nurture  throughout  the  design  process  critical  awareness  of  social  and  cultural  assumptions.  The 
aim  is  the  empowerment  of  participants  to  break  "old  molds"  and  restructure  the  learning  environment  so  that 
ongoing  dialogue  and  critique  are  encouraged.  Students,  teachers  and  designers  alike  should  be  enabled  to  design 
their  own  learning.  Complete  membership  in  another’s  environment  is  integral  and  fundamental  to  the  emancipatory 
design  approach. 

Changes  in  work  activity  as  indicated  by  aspects  of  the  designer's  deployment  of  expertise  in  a group  context 

The  table  below  provides  a brief  overview  of  the  historical  types  in  terms  of  five  aspects  of  any  relationship 
the  designer  is  likely  to  enter  into  during  the  course  of  instructional  design  projects: 

• the  source  and  nature  of  the  expertise  the  designer  employs  (e.g.  technical/discipline  knowledge,  pragmatic,  interpersonal); 

• who  has  the  responsibility/is  accountable  for  doing  the  designing  (including  development,  evaluation,  and 
implementation); 

• the  types  of  conceptual  tools  used  and  when  they  are  used; 

• the  predominant  orientation  of  designer  toward  product,  process,  or  context; 

• the  ethical  principles  (the  "oughts"  and  "shoulds")  guiding  the  designer’s  actions  at  any  given  moment. 

Table  2:  Changes  in  ID  work  activity  as  indicated  by  aspects  of  the  designer's  deployment  of  expertise  in  a 
group  context 


Historical  type 

Nature  of  expertise 

Focus  of 

responsibility  for 
designing 

Tools 

Orientation 

Ethical  Principles 
guiding  designer's 
actions 

Craft  activity 

artistic,  technical 

designer 

intuition, 
experience; 
experimental  design 
principles;  holistic 
evaluation 

product 

work  for  hire, 
produce  results 
contracted  for 

Usability  /User-centered 

technical  tempered 
by  pragmatic 

designer  (users  are 
inputs  to  designer- 
led  process) 

qualitative  research 
methods;  iterative 
process  model; 
holistic  evaluation 

product,  process 

observe  protocols  for 
protecting  human 
subjects;  place  user 
needs/wishes  first 

Rapid  prototyping 

pragmatic,  technical 

designer  (user 
involvement  to 
inform  designer-led 
process) 

"analysis  by  doing" 
(analysis  by 
synthesis");  iterative 
process  model; 
holistic  evaluation 

process,  product 

place  user 
needs/wishes  first; 
suspend 
commitment, 
withhold  judgment 

Socio-tec  hnical 

interpersonal, 
pragmatic,  technical 

shared  among 
designers  and 
stakeholders 
(including  users) 

variety  of  tools  (see 
above);  facilitative 
strategies  and  tactics 

process;  context 

see  all  of  above 

Emancipatory  design 

interpersonal, 
pragmatic,  technical 

shifted  to  those 
"designed  for"; 
user's  are  inputs  to 
their  own  process 

facilitative  strategies 
and  tactics 

process;  context 

moral  obligation  to 
empower  others 

E.  General  observations 

Three  prominent  historical  categories 

In  this  case  study  three  historical  categories  described  the  majority  of  the  work  activity.  Those  categories 
were:  craft  activity,  systematic  design,  and  usability.  Historical  differences  in  the  source  and  in  the  nature  of  the 
expertise  used  and  in  the  tools  employed  in  craft  activity  and  systematic  design  activity  created  disturbances  and 
required  participants  to  negotiate  and  invent. 

The  teacher's  previous  design  and  delivery  of  the  entrepreneurship  course  had  been  conducted  as  a craft 
activity.  The  teacher  was  the  "honcho"  (the  Dean's  characterization  of  his  general  management  philosophy)  of  the 
course,  operating  within  a bureaucracy  that  left  him  alone  as  long  as  teaching  and  learning  outcome  measures  were 
within  acceptable  limits.  The  technology  itself  and  the  increased  visibility  that  technology  drew  to  the  course 
required  the  teacher  to  acquire  and  manage  a greater  variety  of  resources.  With  the  management  requirement  and  the 
visibility  came  accountability,  and  the  need  to  be  more  systematic  about  how  the  course  was  to  be  planned  and 
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executed.  Disturbances  therefore  arose  because  of  changes  in  how  the  work  activity  had  historically  been  organized 
(an  individually  directed  craft  activity)  and  because  of  an  increase  in  its  complexity. 

At  the  beginning  of  the  project  the  teacher  had  bemoaned  his  additional  role  (he  was  instructional  designer 
and  teacher,  as  well)  of  "project  manager"  and  the  complexity  of  orchestrating  the  work  of  8 or  9 people.  By  the  end 
of  the  project,  he  was  relieved  that  in  reality  he  worked  with  far  fewer  individuals.  Concurrently,  Bob  resisted  any 
attempts  at  systematizing  his  design  process  and  his  implementation  of  the  course  (except  for  the  formal  evaluation 
component).  Illustrated  here  are  some  classic  philosophical  notions.  One  is  the  romantic  notion  of  the  "individual  as 
the  source  and  purveyor  of  creativity"  (Coyne  & Snodgrass,  1993),  exemplified  in  the  characterization  of  design  as  a 
craft  activity  carried  out  using  artistic  expertise.  Another  is  the  equally  romantic  notion  of  the  individual  as  rational 
and  objective,  rendered  so  through  the  use  of  logical  analysis  and  its  tools  (e.g.,  explicit  models  of  reasoning). 
Underlying  these  notions  is  the  classical  conflict  between  the  belief  in  rationalism  as  a route  to  gaining  and  using 
knowledge  and  the  belief  in  pragmatism  as  a means  to  the  same  ends. 

Traces  of  past  and  future  conceptions  of  work 

Engestrom  (1987,  1993)  has  suggested  that  the  coexistence  of  various  stages  of  development  in  a work 
activity  create  conflict  and  potential  for  further  development.  The  frequency  of  speaking  "scenarios  of  use"  among 
technologists  in  this  case  study  might  be  considered  an  indication  of  the  influence  of  the  user-centered  design 
movement  on  the  day-to-day  practice  of  instructional  technology  consulting  and  instructional  design.  In  some  cases 
the  scenarios  in  this  project  facilitated  hypothetical  usability  testing.  The  use  of  narratives  in  which  students  and 
teachers  were  the  main  characters  also  indicated  the  relative  importance  the  speaker  gave  to  students  and  teachers  in 
design  activities.  The  teacher  and  the  TLTL  consultant  spontaneously  generated  the  most  scenarios,  consistent  with 
their  social  and  pragmatic  orientation  toward  the  object  of  their  work.  Although  the  independent  technology 
consultant  was  the  self-proclaimed  champion  of  the  user,  he  did  not  seem  as  successful  in  creating  narratives  of  use. 
He  approached  user-centered  design  from  the  "usability  engineering"  perspective,  in  which  the  users  are  necessary 
inputs  or  variables  to  consider,  reflecting  a rationalistic  orientation. 

The  methods,  practices,  and  orientations  characteristic  of  the  usability  stage  of  development  are  a bridge  to 
"humanized"  stages  of  the  development  of  instructional  design,  such  as  participatory  design,  and  to  "emancipatory 
design."  A technical  orientation  toward  usability  testing  and  user-centered  design  is  at  this  point  in  time  well 
ensconced  in  the  theory  and  practice  of  instructional  software  design  and  instructional  design  training.  The  flexibility 
of  the  scenario  or  narrative  in  representing  different  orientations  toward  the  object  of  work  and  its  ability  to 
incorporate  the  salient  aspects  of  people  acting  in  a socio-technical  environment  make  the  scenario  an  important, 
multi-edged  design  tool.  For  example,  John  Carroll’s  investigations  (Chin,  Rosson,  & Carroll,  1997;  Carroll,  1994) 
of  creating  scenarios  of  use  with  users  early  in  the  design  process  expands  the  purpose  of  the  scenario  beyond 
capturing  input  for  designers  on  functionality  and  requirements  to  the  co-designing  of  new  work  activities.  Suchman 
(1995)  pushes  the  scenario's  role  in  user-centered,  participatory  design  even  farther.  When  scenarios  are  preserved 
on  videotape,  for  example,  additional  propositions  arise  connected  to  the  user’s  and  the  designer’s  voice,  authority, 
and  will.  Suchman  suggests: 

• . . .the  aim  of  representing  work  is  to  provide  workers/users  with  a richer  presence  or  stronger  voice  in  sites  of  professional 
design; 

• representation  involves  the  artful  crafting  of  people’s  stories; 

• workers'  perspectives  are  made  available,  present  through  representations,  they  speak  in  their  own  voices; 

• representations  may  become  resources  for  workers’  own  use  in  negotiations  with  management;  and 

• representations  work  against  automation  based  on  simplified  notions  of  work,  (p.60) 

The  concepts  of  usability  and  user-centered  design,  expressed  through  the  scenario  of  use,  or  user-  and 
designer-created  narratives,  can  be  viewed  as  a bridge  from  previous  developmental  forms  of  design  work  activity, 
such  as  craft  and  rationalized  work  forms,  to  humanized  and  emancipatory  forms.  In  this  case  study,  the  teacher  most 
frequently  made  statements  that  exhibited  the  pragmatic  and  social  characteristics  of  historical  types  in  the  lower  part 
of  the  matrix.  The  teacher  could  be  seen,  by  setting  an  example,  as  leading  the  way  across  the  bridge.  Had 
participants  been  more  conscious  of  the  role  that  narrative  could  play  in  their  development  work,  the  bridge  might 
have  been  crossed. 

None  of  these  points  reveal  anything  new  about  our  knowledge  of  people  engaged  in  design  process,  or  the 
way  that  teachers  work  when  designing  and  delivering  courses.  However,  the  matrix  provides  a different  vantage 
point  and  a way  of  articulating  knowledge  about  this  case  of  instructional  design  that  puts  what  we  know  in  a 
context— a historical  one— that  is  not  frequently  examined  in  case  studies  of  this  kind. 

VI.  Implications  for  practice,  research,  and  design  education 

The  findings  and  conclusions  of  this  case  study  have  implications  for  instructional  design  practice,  the 
conduct  and  presentation  of  research  on  design  practice,  and  design  education. 
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Implications  for  practice 

The  method  and  research  perspective  used  in  this  study  provide  simultaneously  a diagnosis  of  what  is  wrong 
and  a suggestion  as  to  what  to  do.  In  this  study  I observed  how  people  work  through  the  disturbances  and  deal  with 
the  systemic  contradictions  underlying  them.  I saw  potentials  for  future  development  of  the  work  activity  of 
instructional  design  emerge  from  the  people  performing  the  activity  in  context.  Future  development  of  instructional 
design  as  a work  activity  would  be  affected  by  acknowledgement  of: 

• the  importance  of  “designing  the  designing”,  in  part  by  coming  to  know  the  cultural  rules,  tools  and  goals  of  collaborators 
and  other  participants,  and  by  understanding  the  communicative  quality  of  one’s  choice  of  rules,  tools,  and  goals; 

• the  pre-eminent  importance  of  an  ethic  of  care  in  the  design  and  implementation  of  instruction; 

• the  importance  of  being  able  to  understand  and  use  narrative  in  instructional  design  work; 

Implications  for  research  on  practice 

I have  demonstrated  that  a coherent  account  of  practice  can  be  constructed  which  includes  and  focuses  on 
social,  cultural,  and  historical  factors.  Using  the  nodes  of  the  activity  system  model  I still  risked  shattering  the 
experience  into  a million  pieces.  Two  key  factors  acted  as  the  “glue”  to  keep  the  analysis  whole:  the  focus  of  the 
analysis  on  interaction  of  the  Subject  with  the  Object  and  its  Outcome,  and  the  analysis  was  guided  by  disturbances, 
which  are  evidence  of  interaction. 

The  historical  analysis  presented  here  suggests  a way  of  tracking  the  influences  of  the  historical 
development  of  the  practice  of  instructional  design  on  day  to  day  design  work.  An  historical  analysis  allows  the 
practitioner  to  understand  that  the  values  and  assumptions  of  a community  of  practice  arise  from  the  accumulated 
experience  over  time  of  individuals  accommodating  contradictions,  and  therefore  that  they  are  constantly  in  a state  of 
change.  Such  a view  of  one’s  role  in  the  community  and  the  role  of  community  in  one’s  work  engenders,  I believe,  a 
greater  acceptance  of  conflict  and  change  in  one’s  immediate  experience,  and  provides  an  impetus  toward  greater 
flexibility  in  the  application  of  methods  and  techniques.  Such  an  attitude  is  important  to  inculcate  in  future 
practitioners  if  instructional  design  field  is  to  have  future  viability. 

Implications  for  design  education 

The  activity  system  model  provides  an  alternative  model  of  design  process  that  incorporates  the  social 
factors  without  the  usual  “mess”  and  without  isolating  them  in  order  to  deal  with  them.  It  is  a model  that  can 
accommodate  the  variety  of  perspectives  instructional  design  researchers  write  about,  and  most  importantly,  can 
represent  and  help  resolve  the  tensions  that  practitioners  experience  in  their  everyday  work.  An  understanding  of 
instructional  design  work  built  around  a more  inclusive,  more  richly  explanatory  model  is  important  in  representing 
instructional  design  work  to  those  who  work  with  instructional  designers  (e.g.  clients,  content  experts,  teachers).  It  is 
important  in  the  education  of  new  designers,  whose  most  earnest  questions  often  have  to  do  with  how  to  do  this  work 
with  others  in  real  contexts. 

Lastly,  and  most  importantly,  the  education  of  future  instructional  designers  must  be  more  fully  and 
explicitly  understood  as  acculturation  because  the  values  and  assumptions  of  the  community  of  practice  are  the  most 
powerful  influences  on  individuals  acting  in  a group  context. 
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Abstract 

Learners  in  distance  education  courses  often  drop  out  because  they  lack  the  motivation  to  go  on.  A study 
which  aimed  at  enhancing  existing  student  support  with  motivational  guidance  in  the  form  of  motivational 
communications,  the  Motivational  Messages  Support  System  (MMSS),  resulted  in  completion  rates  that 
more  than  doubled.  In  the  MMSS  study  developmental  research  has  been  used  for  the  evolutionary 
development  of  the  motivational  communications  and  for  addressing  questions  on  the  validity,  practicality 
and  effectiveness  of  the  motivational  messages. 

Introduction 

More  and  more  learners  are  studying  via  distance  education.  Although  a number  of  students  are  successful 
and  complete  their  courses,  there  are  still  many  students  who  drop  out  or  do  not  complete  their  courses.  Research  has 
shown  that  lack  of  motivation  is  an  important  cause  for  dropout  (Wolcott  & Burnham.  1991;  Zvacek,  1991). 

Contrary  to  conventional  teaching/leaming  situations  in  which  motivational  problems  can  be,  and  often  are,  detected 
by  a teacher  or  trainer,  the  motivational  problems  of  distance  education  students  mostly  go  undetected  and  there  is 
thus  no  outside  help  for  the  students  to  increase  their  motivation.  A way  to  help  distance  education  students  to 
become  or  stay  motivated  is  to  include  motivational  strategies  in  the  learning  materials.  This  was  not  feasible  in  the 
courses  studied,  as  course  updating,  revision  and/or  rewriting  is  normally  only  done  once  every  five  to  ten  years  and 
is  very  costly.  Another  possibility  to  help  learners  to  become  or  stay  motivated  is  extending  existing  student  support 
with  motivational  strategies.  It  is  recognized  that  being  motivated  is  an  autonomous  disposition,  something  that  is 
done  by  the  interested  person  him-or  herself  (J.  Visser,  1990).  Thus  an  ‘outsider’  can  only  intervene  in  motivational 
processes  as  far  as  the  other  person  will  let  it  happen.  It  is  therefore  a challenge  to  develop  a student  support 
component  that  aims  at  improving  the  motivation  of  distance  education  students  so  that  they  finish  distance 
education  courses  successfully.  This  challenge  is  even  greater  when  one  tries  to  support  the  many  distance  education 
learners,  mainly  living  and  working  in  developing  countries,  who  still  study  without  having  access  to  fax  and  or  e- 
mail. 

The  study  aimed  at  introducing  motivational  strategies  in  the  student  support  system  of  a distance  education 
program  that  prepares  international  students  for  a Diploma  or  MA  in  Distance  Education.  It  is  offered  by  the 
University  of  London  and  implemented  by  the  International  Extension  College  (IEC)  in  Cambridge  (England).  The 
students  are  accepted  on  a yearly  basis;  about  50-70%  of  them  comes  from  developing  countries.  The  limited  access 
these  students  have  to  e-mail  and  fax  makes  that  the  courses,  which  are  still  print  based  and  enriched  by  audio 
cassettes,  are  attractive  for  them.  To  enroll  in  the  MA  program  students  first  have  to  finish  the  four  Diploma 
(Foundation)  Courses  of  200  hours  each.  The  actual  MA  part  consists  of  two  courses  and  a research  study.  Course 
completion  rates  are  around  35%  in  the  Foundation  Program.  Only  a handful  of  students  continue  with  the  MA  part. 
Student  support,  in  the  form  of  tutoring,  consists  of  feedback  on  a minimum  of  two  and  a maximum  of  four 
assignments.  No  other  institutionalized  motivational  student  support  is  provided.  The  duration  of  each  course  is  eight 
months. 
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Research  question 

The  study,  which  was  carried  out  in  1996  and  1997  (L.  Visser,  1998),  aims  at  extending  existing  student 
support  with  motivational  strategies  in  the  form  of  motivational  communications  which  should  serve  to  help  distance 
education  students  to  become  or  stay  motivated,  so  that  they  will  complete  their  courses  successfully.  The  study 
seeks  to  answer  the  following  question: 

Is  it  possible  to  design  and  develop  a motivational  component,  in  the  form  of  motivational  communications,  which 
will  result  in  a significant  increase  in  the  percentage  of  students  who  finish  the  course  successfully? 

Review  of  the  literature 

Briggs  (1980)  is  one  of  the  authors  who  emphasize  the  important  role  motivation  plays  in  the  successful 
completion  of  courses.  He  is  of  the  opinion  that  our  theories  or  models  of  design  do  not  sufficiently  take  motivation 
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into  account.  According  to  Keller  (1998)  motivation  is  one  of  the  main  influences  on  performance.  Zvacek  (1991), 
referring  to  distance  education,  observes  that  the  role  of  motivation  cannot  be  overstated,  while  Moore  and  Kearsley 
(1996)  mention  that  tutors  in  distance  education  have  to  pay  a lot  of  attention  to  the  motivation  of  their  students. 

Graham  and  Weiner  (1996)  define  the  study  of  motivation  as  the  study  of  why  people  think  and  behave  as 
they  do.  Motivation  does  not  refer  to  what  a person  can  do,  but  what  a person  will  do;  it  refers  to  choices  people 
make  as  to  what  goals  they  will  set  and  the  degree  of  effort  they  will  exert  in  that  respect  (Keller  (1983). 

There  exists  a number  of  ways  to  change  the  motivational  disposition  of  a learner.  Motivation  can  be  looked 
upon  as  a general  state  necessary  for  learning  to  be  effective.  This  approach  assumes  that  learners  can  create  their 
own  motivational  state.  Another  approach  is  that  motivation  is  a set  of  continually  changing  factors  that  influence  the 
behavior  of  the  learner.  This  approach  aims  at  setting  the  motivational  factors  at  appropriate  levels  so  that  optimal 
learning  can  take  place  (Keller,  1983;  Wlodkowsky,  1985).  There  is  evidence  that  both  approaches  have  positive 
effects  and  the  two  should  probably  be  seen  as  complementary  rather  than  as  opposing  (J.  Visser,  1990). 
Wlodkowsky’s  Time  Continuum  Model  (1985)  and  Keller’s  ARCS  Model  of  Motivational  Analysis  (1983)  analyze 
the  motivational  needs  of  the  learners  in  a number  of  different  components.  Wlodkowsky  uses  an  approach  that 
requires  that  motivational  strategies  must  be  planned  for  each  of  the  six  components  of  the  model  in  order  to  come  to 
a continuous  and  motivational  dynamic.  Wlodkowsky’s  model  is  primarily  prescriptive,  as  it  defines  conditions  and 
associated  learning  situations.  Keller’s  ARCS  model  analyzes  the  motivational  needs  of  learners  in  four  components: 
Attention,  Relevance,  Confidence,  Satisfaction.  To  use  the  model  successfully  an  audience  analysis  has  to  be  done  to 
define  motivational  objectives.  The  motivational  objectives  will  lead  to  motivational  strategies.  The  ARCS  model  is 
geared  towards  a systematic  approach  to  motivational  interventions,  which  follows  a general  problem  solving  and 
design  process  (Keller,  1987).  BOhlin  (1987)  stresses  that  the  ARCS  model  is  adaptable  to  a wide  variety  of  teaching 
methods  and  processes.  He  claims  that  the  model  is  especially  vital  in  situations  where  an  extreme  variance  in 
classroom  processes  is  encouraged  as  it  makes  it  possible  to  accommodate  a wide  range  of  learning  styles  and  to 
develop  individuality.  The  students  of  the  Diploma  and  MA  Courses  of  the  University  of  London  come  from  a 
variety  of  social  and  cultural  backgrounds.  They  also  have  varying  educational  backgrounds  and  use  different 
learning  strategies  in  a distance  education  context.  The  systematic  approach,  which  is  an  asset  of  the  ARCS  model, 
gives  the  distance  education  tutor8  a way  to  monitor  the  motivational  state  of  the  learners.  As  far  as  known  neither 
Wlodkowsky’s  model  nor  Keller’s  model  has  been  used  in  a distance  education  support  system.  Keller’s  ARCS 
model  has,  however,  formed  the  basis  for  the  development  of  motivational  communications  in  traditional  education, 
which  were  successfully  used  to  help  learners  to  maintain  or  increase  their  motivation  (J.  Visser,  1990).  The 
flexibility,  the  systematic  approach  and  the  fact  that  the  model  has  earlier  been  successfully  used  in  a comparable 
context,  has  led  to  the  decision  to  opt  for  the  ARCS  model  of  motivational  analysis  in  the  MMSS. 


The  ARCS  model  of  motivational  analysis 

The  four  components  of  the  ARCS  model  (Keller,  1983)  represent  the  most  important  dimensions  of  human 
motivation.  They  are  briefly  described  and  placed  in  the  context  of  a distance  education  setting. 


Attention: 


Relevance: 


Confidence: 


Satisfaction: 


Getting  the  attention  of  the  student  and  getting  the  learner  to  focus  on  a learning  task  establishes  the  necessary 
conditions  to  leam.  This  is  likely  to  be  more  difficult  in  distance  education  than  in  conventional  education.  A 
challenging,  unexpected  or  even  provocative  message  will  stimulate  the  attention  of  the  learner.  Other 
motivational  strategies  include  frequent  communication,  rapid  feedback  and  real  interest  in  the  learner. 

Students  should  be  informed  on  what  they  can  expect  from  a course  and  why  the  topic  in  question  is  relevant 
and  relates  to  the  larger  scheme  of  a course.  Just  like  in  conventional  education,  distance  education  students 
should  know  why  they  study  certain  course  sections  and  they  should  be  given  examples  related  to  their  daily 
work.  Relevance  is  not  expected  to  be  a problem  for  the  students  of  the  courses  taking  part  in  the  MMSS  study, 
as  learners  are  mostly  working  in  distance  education. 

Rotter  (1954)  states  that  people  have  a tendency  to  attribute  success  and  failures  to  external  or  internal  causes. 
Following  success,  expectancy  generally  rises  (confidence  increases),  while  failure  causes  confidence  to  drop. 

In  distance  education  fear  of  failure  is  generally  high.  Messages  emphasizing  that  students  can  finish  the  course 
successfully  if  they  work  hard,  are  important.  Providing  frequent  opportunities  for  students  to  be  successful 
increases  the  chance  that  they  really  will  be  successful.  Emphasizing  that  students,  themselves,  are  responsible 
for  success  through  their  personal  ability  and  effort  builds  confidence. 

To  allow  a student  to  be  satisfied,  it  is  important  to  describe  the  task  to  be  done  carefully  and  to  state  the 
reward  upon  completion.  Keller  (1987)  claims  that  effort  rather  than  performance  is  most  directly  related  to 
motivation.  In  distance  education  student  support  motivational  tactics  should  aim  at  making  students  feel 
satisfied  frequently.  Frequent,  timely,  adequate  and  encouraging  feedback  is  an  important  satisfaction  strategy. 
Other  strategies  are  personal  praising  remarks  and  informing  the  student  on  how  far  they  have  come  already. 


8 In  the  context  of  this  paper  a tutor  is  the  person  who  academically  accompanies  the  distance  education  student 
during  the  course  period. 
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Keller's  (1987)  “Steps  in  Motivational  Design”,  which  is  based  on  the  ARCS  model,  has  been  used  for  the 
design  and  development  of  the  so-called  motivational  communications  developed  in  this  study  and  used  in  the 
Motivational  Messages  Support  System  (MMSS). 

The  Motivational  Messages  Support  System 

The  MMSS  had  to  be  designed  recognizing  that  it  had  to  function  as  a subsystem  within  the  course  delivery 
system  the  International  Extension  College  has  set  up  to  provide  distance  education.  Before  the  MMSS  was 
introduced  there  was  limited  interaction  between  the  various  components  (tutors  and  students)  of  the  support  system 
and  there  was  not  institutionalized  direct  communication  between  student  and  tutor  and  vice  versa:  all 
communication  was  routed  via  the  Director  of  Courses.  It  was  concluded  that,  if  the  ARCS  model  was  going  to  be 
used,  establishing  the  possibility  for  direct  communication  between  student  and  tutor  was  a necessary  condition  for 
enhancing  students'  motivation.  A motivational  component  was  added  to  the  existing  student  support  system  in  the 
form  of  a structured  set  of  written  motivational  communications.  These  communications,  or  motivational  messages 
as  they  are  called  in  the  study,  specifically  target  the  motivational  disposition  of  the  learners  so  that  they  finish  the 
course  successfully. 

Research  approach 

Against  the  above  sketched  background  a developmental  research  approach  was  chosen.  Developmental 
research  is  carried  out  to  optimize  and  gain  a sound  basis  for  development  activities,  particularly  the  development  of 
prototypical  products  and  the  generation  of  methodological  directions  for  the  development  of  such  products  (Van 
den  Akker  & Plomp,  1993;  Richey  & Nelson,  1996).  The  aim  of  the  study  is  thus,  through  a sequence  of  prototypes, 
to  come  to  a final  quality  product. 

Van  den  Akker  and  Plomp  (1993)  and  Nieveen  (1997)  discussing  product  quality,  distinguish  the  following 
three  criteria: 

Validity:  the  degree  to  which  the  product  is  in  agreement  with  state-of-the  art  knowledge  and  is  internally 

consistent. 

Practicality:  the  degree  to  which  the  product  is  functional  for  the  target  group. 

Effectiveness:  the  degree  to  which  the  product  has  an  added  value  compared  to  already  existing  products. 

The  first  two  criteria,  validity  and  practicality  have  to  be  dealt  with  before  the  effectiveness  of  a product  can 
be  assessed.  In  this  study  validity  was  realized  by  using  a methodical  design  approach  while  building  on  a sound 
theoretical  framework  (Keller,  1987).  Practicality  is  shown  when  students  state  that  they  appreciate  the  motivational 
messages  and  when  tutors  communicate  that  they  can  easily  work  with  the  messages.  Effectiveness  is  shown  when  a 
significant  higher  percentage  of  students  than  the  year  before  complete  the  courses  successfully.  The  study  consisted 
of  two  phases,  a pilot  study  which  aimed  at  developing  the  MMSS  system  and  the  main  study  which  aimed  at 
assessing  the  effectiveness  and  cost-efficiency  of  the  MMSS. 

Pilot  Study 

The  pilot  study  addressed  the  following  question: 

Is  it  possible  to  design  motivational  messages,  based  on  the  ARCS  model  of  motivational  analysis,  in  such  a way 

that  they  are  effective  in  distance  education  courses? 

In  this  phase,  first  motivational  communications  were  to  be  developed  as  prototypical  products  along  the 
lines  of  the  ARCS  model  to  be  used  in  one  course.  A set  of  eight  motivational  messages  was  designed,  based  on  a 
design  process  developed  by  Keller  (1987).  Each  message  had,  as  an  individual  prototype,  a special  function  within 
the  framework  of  the  ARCS  model  and  the  framework  of  prototyping,  as  to  characteristics  of  format,  graphics, 
content  and  students'  feedback.  Each  message  could,  however,  considered  to  be  a prototype  for  the  next  one  in  a 
sequence  of  prototypical  messages  to  be  developed,  which  equally  dealt  with  characteristics  of  format,  graphics, 
content  and  reactions  of  the  students.  The  development  of  the  motivational  messages  was  also  influenced  by  expert 
comments.  All  messages  focussed  on  the  motivational  needs  of  the  learners  at  that  moment  in  time. 

A second  objective  of  the  pilot  phase  was  to  investigate,  through  formative  evaluation,  the  validity  and 
practicality  of  the  motivational  messages  and  to  explore  their  effectiveness.  For  this  phase  one  of  the  Foundation 
Courses  was  selected,  consisting  of  19  students  from  13  countries  (five  continents)  of  which  the  greater  majority 
could  only  be  contacted  via  the  normal  postal  services.  As  a consequence  the  delay  in  the  responses  from  the 
students  to  a certain  message  was  too  long  to  take  their  comments  immediately  into  consideration  when  designing  the 
next  prototype  of  a message  in  the  sequence.  The  feedback  of  the  students  was  therefore  only  used  in  subsequent 
prototypes.  In  addition,  in  the  evaluation  of  the  motivational  messages  expert  appraisal  was  applied  by  involving 
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staff  of  the  college  and  other  experts  in  the  field  of  instructional  design  and  distance  education.  The  validity  and 
practicality  of  the  prototypical  messages  was  measured  by  using  the  following  instruments: 

• Enrolment  forms  providing  information  on  students’  social  and  academic  backgrounds. 

• Final  questionnaires  which  asked  questions  about  appeal,  relevance,  layout  and  format  of  the  messages. 

• Course  completion  records  of  1995  and  1996 

• Records  with  information  about  tutoring  time,  frequency  and  duration  of  contact  and  number  of  communications  sent  out 
and  received. 

• Telephone  interviews  with  the  Director  of  Courses  and  the  Executive  Director  of  the  IEC,  as  to  implementation 
possibilities,  reactions  from  colleague  tutors  and  students  and  discussion  of  topics  like  course  evaluation  questionnaires. 

• Interviews  with  two  students  of  the  course  to  discuss  format,  lay-out,  frequency  and  content  of  the  messages. 

The  effectiveness  of  the  messages  was  explored  by  comparing  the  percentage  of  students  that  passed  the 
exam  at  the  end  of  the  course  year  (1996)  to  that  of  the  year  before  (1995).  Qualitative  remarks  of  the  students  were 
also  taken  into  account. 

The  result  of  the  pilot  study  has  been  the  Motivational  Message  Support  System  (MMSS)  ready  to  be 
assessed  for  its  effectiveness  and  cost-efficiency.  It  was  noticed  that  designing,  producing  and  personalizing  the 
messages  had  taken  quite  some  time.  As  student  support  is  already  considered  an  expensive  commodity  in  distance 
education  (Bates,  1991;  Threlkeld  & Brzoska,  1994),  it  was  decided,  on  the  basis  of  the  pilot  study,  to  use  two 
different  versions  of  the  motivational  messages  in  the  main  study.  One  version  used  the  same  set  of  messages  for  all 
students  (collective  messages),  while  in  the  other  version  the  tutor  could  personalize  the  messages  by  adding 
personal  remarks  to  the  ‘standard’  messages  or  design  messages  that  were  attending  to  the  perceived  motivational 
needs  of  the  students  (personalized  messages). 

The  assessment  phase 

The  main  study  addressed  two  questions. 

To  what  extent  are  motivational  messages,  based  on  Keller’s  ARCS  model  of  motivational  analysis,  effective  in 
distance  education  courses? 

What  is  the  difference,  if  any,  in  effectiveness  and  cost-efficiency  between  the  personalized  and  the  collective 
process  of  enhancing  motivation  through  motivational  messages? 

The  first  research  question,  which  had  already  been  explored  in  the  pilot  study,  focuses  mainly  on  the 
validity,  practicality  and  effectiveness  of  the  motivational  messages.  The  second  research  question  looks  at  the 
optimal  use  of  the  motivational  messages  and  has  an  exploratory  character. 

Effectiveness  in  the  MMSS  study  is  shown  when  a significant  higher  percentage  of  students  than  the  years 
before  complete  the  courses  successfully.  Cost-efficiency  is  shown  when  a significant  higher  percentage  of  students 
than  the  years  before  finish  the  courses  and  when  the  time  spent  on  tutoring  (tutor  costs)  and  overhead  costs  does  not 
significantly  increase.  Cost-efficiency  relates  to  the  outputs  realized  as  compared  to  the  corresponding  inputs. 

It  is  likely  that  implementing  the  MMSS  will  increase  the  tutor  costs  in  terms  of  hours  spent  on  a student. 
Eight  hours  of  tutoring  time  are  budgeted  for  per  student  per  course.  Students  who  repeat  a course  can  be  expensive 
for  the  International  Extension  College  as  they  may  not  only  require  extra  tutoring  time,  but  also  extra  time  related  to 
registering  procedures. 

For  the  main  study  a multiple  case  design  was  chosen  (Yin,  1994).  As  shown  in  Table  1,  all  four  courses  of 
the  Foundation  Program  were  involved.  Course  B was  split  into  two  groups  and  Course  A served  as  a control  group: 
the  tutor  of  that  course  was  only  to  write  down  the  time  spent  on  tutoring  according  to  the  tutor  contract. 
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Table  1:  Overview  of  the  courses  involved  in  the  MMSS  study  and  the  number  of  messages  to  be  sent  out. 


Course 

code 

# students 

Type  Message 

# messages 

Designer 

messages 

Tutor 

A 

18 

No  messages 

N/A. 

N/A. 

External 

B1 

13 

Collective 

8 

Course  tutor 

External  (researcher) 

B2 

14 

Personalized 

To  be  decided  by 
tutor 

Course  tutor 

External  (researcher) 

C 

18 

Collective 

8 

Researcher  and 
tutor 

Staff-member  College 

D 

18 

Personalized 

To  be  decided  by 
tutor 

Course  tutor 

External 

In  those  courses  where  personalized  messages  were  used,  it  was  the  tutor  who  decided  on  how  many 
messages  would  be  sent  out,  as  the  messages  were  based  on  the  perceived  motivational  needs  of  the  students  during 
the  course.  The  tutors  in  Courses  B2  and  D decided  beforehand  that  a minimum  of  four  messages  would  be  sent  out. 
In  order  to  establish  habits  and  conditions  for  independent  learning,  the  interval  in  time  between  the  motivational 
messages  became  longer.  When  the  main  study  was  finished  and  collaborating  tutors  had  to  send  their  data  to  the 
researcher  it  turned  out  that  the  tutor  of  Course  D,  who  was  to  use  personalized  messages,  had  only  sent  out  one 
message.  The  MMSS  had  thus  not  functioned  in  Course  D. 


A variety  of  instruments  were  used  in  the  MMSS  main  study. 

Student  related  instruments 

• Initial  questionnaires  aimed  at  getting  information  on: 

• Students’  personal  and  professional  circumstances. 

• Students'  reasons  for  taking  the  course 

• Expectations  of  students  as  to  support  from  IEC  and  tutors 

• Problems  students  expected  while  doing  the  courses. 

• Final  questionnaires  aimed  at  getting  information  on: 

• Which  messages  students  liked  best  and  why. 

• Which  messages  were  most  relevant  to  students  and  why. 

• What  the  students  thought  the  reason  was  for  receiving  the  messages. 

• Telephone  interviews  with  two  students  (a  reluctant  enrollee  and  a repeater)  to  see  how  the  messages  were  appreciated  and  if  so, 
why. 


The  term  'motivational  message'  was  never  used  in  correspondence  with  the  students,  as  this  could  have 
revealed  the  intended  nature  of  the  messages. 

Tutor  related  instruments: 

• Course  completion  records  of  1995,  1996  and  1997 

• Records  to  register  marking  and  tutoring  time,  number  and  content  of  communications  between  tutor  and  student  and  vice 
versa. 

• Telephone  interviews  with  tutors  and  expert  staff. 

• Structured  interviews  with  tutors. 

While  developing  the  instruments,  an  important  concern  had  been  to  make  sure  that  triangulation  would  be 
possible.  Various  methods  of  data  collection  were  used  which  aimed  at  enhancing  the  reliability  and  the  validity  of 
the  findings.  In  addition  the  researcher  had  designed  a tutor  guide  to  make  sure  that  tutors  could  work  independently 
of  the  researcher  during  the  main  study. 

The  MMSS  pilot  study 

The  principal  aim  of  the  pilot  study  was  to  come  to  a valid,  practical  and  effective  version  of  the 
motivational  messages  as  well  as  to  answer  questions  on  design  and  distribution.  During  the  prototyping  phase  the 
development  of  the  product  was  carefully  monitored.  Feedback  and  comments  (solicited  and  unsolicited)  were  used 
to  improve  the  product  and  its  distribution  process.  The  research  lasted  eight  months  and  aimed  at  answering  the 
following  question: 
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Is  it  possible  to  design  motivational  messages,  based  on  the  ARCS  model  of  motivational  analysis,  in  such  a way 
that  they  are  effective  in  distance  education  courses? 

During  the  first  month  of  the  course  a motivational  needs  profile  was  made  which  was  based  on  a limited 
audience  analysis  (the  names  and  addresses  of  the  students  only  became  available  when  the  course  had  already 
started),  on  information  provided  by  the  student  counselor  and  on  research  in  the  field  of  student  support.  During  the 
courses,  when  more  information  became  available  about  the  motivational  needs  of  the  students,  this  profile  was 
adjusted.  A total  of  eight  messages  was  sent  out,  which  were  often  personalized  by  a remark  or  an  observation  that 
targeted  the  perceived  motivational  needs  of  the  student.  It  was  observed  that  qua  format  students  liked  motivational 
messages  in  the  form  of  a greeting  card  best,  qua  content  they  preferred  motivational  messages  that  had  a catching 
drawing  on  the  outside  and  a reinforcing  text  on  the  inside.  The  last  message  they  received  was,  according  to  the 
final  evaluation  questionnaire,  the  favorite.  It  had  the  format  of  a greeting  card  and  showed  a variety  of  students  of 
different  ages,  gender  and  race  on  the  outside.  The  inside  consisted  of  a short  text  which  stressed  that  they  all 
belonged  to  a group  of  people  working  hard  to  finish  the  courses  successfully.  It  also  contained  exam  advice  and 
stressed  that  students  had  a good  chance  to  do  well  if  they  spend  their  last  few  weeks  studying  hard. 

In  summary  it  can  be  said  that: 

The  students  testified  that  the  communications  had  motivated  them. 

The  students  reported  that  the  motivational  messages  helped  them  to  stay  in  the  course. 

The  students  liked  the  last  prototypical  message  best. 


It  is  thus  concluded  that  the  motivational  messages  helped  students  to  stay  in  the  course.  This  conclusion 
was  confirmed  by  the  course  completion  rates  of  1996  which  went  up  from  32%  in  1995  to  53%  in  1996. 

It  was,  however,  also  seen  that  the  use  of  motivational  messages  was  labor  intensive  as  it  involved  doing  an 
audience  analysis,  defining  motivational  objectives,  designing,  producing  and  dispatching  motivational  messages.  In 
addition  the  students’  motivational  needs  had  to  be  monitored.  It  was  also  noticed  that  tutors  needed  some  home 
office  equipment  such  as  a computer,  a printer  and  a photocopier  to  produce  and  send  out  messages.  This  led  to  the 
decision  to  expand  the  study  by  using  a more  standardized  type  of  message  which  was  to  be  prepared  at  the 
beginning  of  the  course  and  based  on  a motivational  needs  analysis  (collective  motivational  messages). 

Further  insights  were  necessary  to  determine  whether  the  motivational  messages  that  were  personalized  or 
custom-made  were  more  effective  than  the  collective  motivational  messages.  In  addition  questions  on  cost-efficiency 
had  to  be  answered. 

Three  different  cases  were  to  be  assessed  in  the  main  study 


• A first  group  of  students  would  receive  personalized  motivational  messages,  based  on  an  audience  analysis,  on  the  results 
of  the  pilot  study  and  on  monitoring  the  motivational  needs  of  the  students. 

• A second  group  of  learners  would  receive  collective  motivational  messages,  based  on  an  initial  audience  analysis  and  on 
the  results  of  the  pilot  study.  These  messages  would  be  prepared  during  the  first  month  of  the  course,  so  that  the  messages 
only  had  to  be  sent  out. 

• A third  group  of  students  would  serve  as  a control  group  and  would  not  receive  the  MMSS  support.  Tutoring  in  this  group 
was  done  according  to  the  tutor  contract  and  time  spent  on  tutoring  was  going  to  be  recorded. 

In  the  above  sketched  set-up  it  would  be  possible  to  decide  which  type  of  motivational  messages  is  more 
effective  and  which  more  cost-efficient. 

The  MMSS  assessment  phase 

The  first  phase  of  the  MMSS,  the  pilot  study,  took  place  in  1996  and  was  followed  by  the  main  study  in 
1997.  Two  questions  were  addressed: 

1 . To  what  extent  are  motivational  messages,  based  on  Keller’s  ARCS  model  of  motivational  analysis,  effective  in  distance  education 
courses? 

2.  What  is  the  difference,  if  any,  in  effectiveness  and  cost-efficiency  between  the  personalized  and  the  collective  process  of  enhancing 
motivation  through  motivational  messages? 


In  Table  1 the  participating  courses  and  the  types  of  messages  used  in  the  MMSS  study  were  given.  Two 
months  before  the  study  took  off  the  participating  tutors  received  a number  of  articles  on  motivation  and  a copy  of 
the  pilot  study,  wile  a month  before  the  courses  started  the  tutors  were  personally  briefed  during  a half-day  session 
and  received  a tutor  guide  containing  information  on  how  to  use  the  motivational  messages  and  how  to  do  the  record 
keeping. 
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The  first  research  question 

The  first  activity  the  tutor  carried  out  was  sending  all  students  an  audience  analysis  questionnaire.  On  the 
basis  of  the  outcomes  of  this  questionnaire  motivational  messages  were  designed,  produced  and,  during  the  course, 
sent  out.  The  tutor  of  Course  C requested  the  researcher’s  assistance  in  designing  the  messages,  but  the  tutor  of 
Course  D worked  on  his  own,  while  the  researcher  in  her  role  as  tutor  also  designed  and  produced  messages  for 
Courses  B 1 and  B2.  Care  was  taken  not  to  interfere  in  the  tutorial  processes.  Tutors  had  been  asked  to  record 
exchanges  they  had  with  their  students,  the  time  spent  on  tutoring  and  remarks  and  details  they  wanted  to 
communicate.  Just  after  the  exams,  but  before  the  results  were  known,  a final  questionnaire  was  sent  to  the  students. 
The  last  involvement  of  the  tutors  in  the  research  consisted  of  returning  the  records  to  the  researcher.  It  was  then  that 
the  researcher  discovered  that  the  tutor  of  course  D had  only  sent  out  one  message.  It  can  thus  be  concluded  that . 
Course  D did  not  participate  in  the  research. 

Due  to  the  scope  of  this  paper  only  a summary  of  the  completion  rates  of  the  students  of  Courses  B and  C is 
given  in  Table  2 (L.  Visser,  1998). 

Table  2:  Completion  rates  Courses  B and  C in  1995,  1996  and  1997.  Bold  figures  indicate  that  motivational 
strategies  were  used. 


1995 

1996 

1997 

Course  B 

# students 

12 

30 

27 

# completors 

4 33% 

11  37% 

18  67% 

Course  C 

# students 

20 

19 

18 

# completers 

6 30% 

10  53% 

11  61% 

The  groups,  which  received  the  motivation  stimulus  have  higher  completion  rates  than  the  groups  that  did 
not  receive  the  stimulus.  Overall,  this  difference  is  statistical  significant  (Fisher’s  Exact  Test,  p=.003).  For  Course  B 
and  C,  the  p-values  from  the  Fisher  Exact  Test  are  respectively  .015  and  .094,  which  means  that  when  a critical  alpha 
of  .05  is  applied,  the  difference  is  only  significant  for  Course  B.  However,  one  might  argue  that  we  still  can  be  quite 
confident  that  the  difference  for  Course  C is  not  due  to  chance  fluctuations. 

It  was  seen  that  students  had  appreciated  the  motivational  messages.  Of  the  45  students  who  had  received 
the  final  questionnaire  25  returned  it.  Of  these  25  students,  22  said  that  the  motivational  messages  had  encouraged 
them  to  continue  with  the  course,  while  23  wrote  that  the  notes  had  contributed  to  their  pleasure 

The  second  research  question 

Students  generally  did  not  perceive  the  personalized  and  collective  motivational  messages  as  being 
different.  It  is  seen  in  Table  3 that  those  students  who  received  collective  messages  had  less  contact  with  their  tutor 
than  those  whom  received  the  personalized  messages. 

Table  3:  Number  of  exchanges  between  tutor  and  student  and  student  and  tutor 


Tutor  student 

Student  tutor 

Personalized  messages 

Average  # exchanges 

3.9 

3.3 

Collective  messages 

Average  # exchanges 

2.6 

2.9 

There  was  a difference  in  the  tutor  time  between  the  two  different  types  of  messages.  For  the  personalized 
motivational  messages  the  tutor  time  (which  did  not  include  feedback  time  on  Tutor  Marked  Assignments)  was  184 
minutes  (Course  B2).  In  Course  B 1 and  Course  C,  which  used  collective  motivational  messages,  the  time  was  148 
minutes.  Collective  messages  were  thus  more  cost-efficient  as  to  tutor  time.  In  two  of  the  three  courses  the  total  time 
spent  on  each  student  stayed  within  the  allocated  8 hours.  In  Course  C it  was  almost  9 hours,  but  it  should  be  noted 
that  the  tutor  of  Course  C was  new  to  the  program. 

The  student  completion  rate  over  1997  is  given  in  Table  4. 

Table  4:  Course  completion  rates  of  students  in  1997 


1997 

Total  # students 

Completed 

Personalized  messages 

14 

1 1 (79%) 

Collective  messages 

31 

18  (58%) 

It  is  noted  that  the  completion  rates  in  1997  were  significantly  higher  for  those  students  who  had  been 
enrolled  for  the  first  time  that  year  than  for  those  who  were  repeaters.  For  personalized  messages  the  completion  rate 

’ 1 

409 


BEST  COPY  AVAILABLE 


388 


was  100%  and  for  collective  messages  it  was  73%.  It  should  be  borne  in  mind  that  the  numbers  are  small,  so  that 
more  research  should  be  done  to  confirm  these  data. 

In  conclusion 

It  is  seen  that  the  outcomes  of  the  main  study  confirmed  the  major  findings  of  the  pilot  study  and  that  the 
messages  have  been  appreciated  by  the  students  and  have  been  effective.  No  significant  difference  between  the  use 
of  collective  and  personalized  messages  as  regards  effectiveness  was  shown.  The  use  of  prepared  messages,  the 
collective  system,  made  it  easier  for  the  tutors  to  use  the  MMSS  and  thus  increased  the  chance  of  successful 
implementation.  The  latter  were  also  more  cost-efficient. 

Although  designing  and  using  the  motivational  messages  is  something  that  is  interesting  and  rewarding,  it 
should  be  seen  as  a motivational  process  and  cannot  be  looked  upon  in  isolation.  Implementing  the  MMSS  means 
that  new  roles  will  have  to  be  taken  on  by  the  tutors.  They  will  no  longer  just  be  ‘markers’  who  give  feedback  on 
assignments  and  write  hints  for  improvement  in  the  margin.  The  tutor  must  become  much  more  personally  involved 
in  the  tutoring  and  then  should  also  be  held  accountable  for  what  happens  in  the  course  in  terms  of  tutor-student 
interaction  and  student  performance.  The  role  of  the  tutor  will  more  match  that  of  a good  teacher  in  conventional 
education  than  that  of  a traditional  tutor  in  distance  education.  This  will  increase  the  tutor  costs  and  has  implications 
for  the  providing  institute. 

The  providing  institute  will  have  to  make  changes  in  two  areas.  The  position  of  the  tutor  should  be 
redefined  as  argued  above.  More  responsibility  should  be  assigned  to  tutors.  The  use  of  modem  communication 
means  should  be  made  possible  and  should  be  encouraged.  This  may  make  it  necessary  to  make  the  selection  of 
tutors  more  rigid.  Such  change  processes  are  not  easy  and  often  somewhat  painful.  Their  effectiveness  depends 
greatly  on  the  extent  to  which  the  institution  is  able  to  learn. 

The  MMSS  study  has  answered  a number  of  questions  related  to  student  support  in  distance  education.  It 
has  been  seen  that  using  motivational  strategies  in  learner  support  has  considerably  increased  the  completion  rates  of 
the  students.  The  MMSS  has  operated  in  a limited  environment  of  only  one  specific  college  and  with  relatively  small 
numbers  of  students  involved.  This  calls  for  more  research.  In  the  study  a case  has  been  made  that  it  is  possible  to 
increase  the  number  of  students  who  finish  their  distance  education  courses  successfully  and  more  in  general  to  make 
the  learning  experience  of  the  students  in  a distance  education  setting  more  enjoyable  and  rewarding.  As  the  learning 
landscape  is  gradually  changing  and  more  and  more  learning  opportunities  are  being  offered  through  the  distance 
education  mode,  it  is  important  that  increased  attention  be  given  to  the  motivational  dimension  of  the  interaction  with 
distance  education  students. 
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(behavior).  Furthermore,  the  principal  dispositional  bottleneck  between  opportunity  and  action  is  sensitivity: 
People  do  not  detect  shortfalls  in  thinking  in  the  first  place. 


Attitudes  and  Spontaneous  vs.  Deliberative  Processing 


How  do  these  findings  pertain  to  learning  and  instruction?  Focussing  on  sensitivity,  we  may  ask,  “How  can 
people  learn  to  identify  occasions  that  call  for  thoughtful  attention?” 

We  wish  to  frame  our  answer  to  this  question  by  addressing  the  similarities  between  dispositions  and 
attitudes.  A disposition  such  as  the  disposition  to  seek  and  evaluate  reasons  can  be  recognized  in  a person  by  that 
person’s  behavior.  A strong,  or  well-developed,  disposition  to  seek  and  evaluate  reasons  is  manifested  in  situations 
where  seeking  and  evaluating  reasons  is  appropriate.  This  cause-effect  relationship  between  disposition  and  action 
points  out  the  similarity  between  dispositions  and  attitudes.  Robert  Gagn6  puts  it  even  more  clearly,  defining  an 
attitude  as  “a  disposition  or  readiness  for  some  kind  of  action”  (1977,  p.  236). 

Through  investigations  into  the  influence  of  attitude  on  action,  Russell  Fazio  and  his  colleagues  have 
demonstrated  that  a person’s  behavior  is  consistent  with  an  attitude  to  the  extent  that  the  attitude  is  strong  and 
therefore  easily  accessible  from  memory  (Fazio,  Blascovich,  & Driscoll,  1992;  Fazio,  Herr,  & Olney,  1984).  These 
researchers  conceive  of  an  attitude  as  an  association  between  an  object  (or  issue,  or  event)  and  an  individual’s 


evaluation  of  the  object.  The  strength  of  this  association  determines  the  likelihood  that  the  attitude  will  be  activated 
from  memory  when  the  individual  again  encounters  the  attitude-inducing  object  (Fazio,  et  al.,  1992). 

According  to  Fazio  (1990),  a person’s  judgments  and  action  can  stem  from  either  of  two  processing 
modes — a spontaneous  process  or  a more  effortful,  deliberative  process.  The  spontaneous  process  requires  that 
one’s  attitude  be  activated  from  memory  by.an  attitude-inducing  object.  The  automatically  activated  attitude  then 
serves  as  a filter  that  influences  subsequent  object-relevant  information  processing,  judgment,  and  action.  In 
contrast,  during  deliberative  processing,  individuals  carefully  examine  the  available  information,  analyzing  the  likely 
consequences  of  options  before  deciding  on  an  intention  to  act  in  a certain  way.  Fazio’s  model  integrates 
spontaneous  and  deliberative  processes  by  postulating  motivation  and  opportunity  as  the  two  factors  that  determine 
which  process  will  occur:  To  the  extent  that  preexisting  attitudes  are  capable  of  automatic  activation,  they  will 
govern  a person’s  attention,  judgments,  and  actions — unless  he  or  she  is  motivated  to  engage  in  deliberative 
processing  and  is  given  sufficient  opportunity  to  do  so. 

In  situations  where  there  is  low  motivation  to  deliberate  over  decisions,  preexisting  attitudes  are  activated 
from  memory.  These  attitudes,  although  often  biased,  free  the  individual  from  the  processing  effort  required  for 
critical  thought.  Fiske  and  Taylor  (1991)  use  the  term  “cognitive  misers”  to  describe  people  exhibiting  this  very 
common  tendency.  In  contrast,  motivated  individuals  can  overcome  the  potentially  biasing  influence  of  a strong, 
preexisting  attitude — if  they  are  given  the  opportunity  to  deliberately  bring  critical  thinking  processes  to  a problem 
situation. 


Attitude  Strength  and  Sensitivity  to  Occasion 

The  contrast  between  spontaneous  processing  and  deliberative  processing  provides  an  explanation  for 
Perkins  and  Tishman’s  conclusion  that  sensitivity  to  occasion  is  the  bottleneck  to  critical  thinking.  When  a person 
does  not  apply  critical  thinking  in  a situation  that  calls  for  it,  this  does  not  necessarily  reflect  a lack  of  ability  or  even 
a lack  of  inclination  to  do  so.  Rather,  it  may  be  the  case  that  the  person  is  a cognitive  miser  in  this  situation, 
exhibiting  spontaneous  behavior  influenced  by  an  automatically  activated  attitude.  The  attitude  activated  is  one  that 
identifies  the  situation  as  not  requiring  any  particularly  deliberate  thought. 

Recognition  of  an  occasion  that  calls  for  thoughtful  attention  may  occur  in  one  of  two  ways.  On  the  one 
hand,  an  individual  may  not  normally  recognize  a situation  that  calls  for  critical  thinking.  On  a particular  occasion, 
however,  she  may  be  motivated  and  have  the  opportunity  to  deliberate  over  the  situation.  Upon  examining  the 
options  available,  the  individual’s  inclination  and  ability  to  make  thoughtful  choices  then  come  into  play.  On  the 
other  hand,  a person  may  have  already  acquired  a strong  attitude  of  critical  thinking  in  certain  situations.  On  such 
situations,  this  attitude  will  be  spontaneously  activated  and  the  person  will  recognize  the  need  for  thoughtful 
attention.  She  will  then  most  likely  be  inclined  to  act  thoughtfully  and  carry  out  her  intentions. 

Strong  and  accessible  attitudes  appear  to  have  a functional  value  in  that  they  serve  to  direct  our  attention  to 
certain  objects.  Roskos-Ewoldsend  and  Fazio  (1992)  conducted  a series  of  experiments  in  which  they  demonstrated 
that  an  individual’s  attention  is  automatically  drawn  to  attitude-activating  objects,  those  objects  toward  which  the 
individual  has  a strongly  associated  evaluation  in  memory.  An  attentional  component  of  attitudinal  memory  was 
identified: 

Collectively,  these  findings  demonstrate  that  the  extent  to  which  objects  attract  attention  as  they  enter  our 
visual  field  depends,  at  least  in  part,  on  the  accessibility  of  our  attitudes  toward  the  objects.  If  a strongly  associated 
evaluation  of  the  object  exists  in  memory,  then  that  object  attracts  attention.  Hence,  we  are  likely  to  notice  those 
objects  that  we  have  personally  defined  as  likeable — those  that  can  provide  some  reward  or  satisfaction — those  that 
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From  Ability  to  Action:  Designing  Instruction  for  Critical  Thinking  Dispositions 
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Abstract 

Perkins  and  Tishman  (1998)  found  that  people  often  do  not  detect  the  need  for  critical  thinking  even 
though  they  may  be  inclined  to  do  so  and  have  the  ability.  This  dispositional  aspect  of  critical  thinking 
may  be  considered  part  of  altitudinal  memory , readily  activated  if  sufficiently  strong.  We  suggest  that  ill- 
structured  problem  solving  can  provide  learners  with  motivating  activities  that  strengthen  critical  thinking 
dispositions.  The  Jasper  Series  and  Decision  Making  are  reviewed  as  examples  of  instructional  programs 
that  incorporate  important  attitude-strengthening  elements  (Fazio,  1995)  into  such  activities.  Design 
principles  for  nurturance  of  critical  thinking  dispositions  are  also  presented. 

Introduction 

Thinking  critically  in  problem  situations  is  a skill  that  has  received  increasing  attention  as  an  educational 
goal.  Critical  thinking  is  often  described  as  an  ability.  Yet,  ability  is  only  one  aspect  of  critical  thinking.  People 
behave  more  or  less  intelligently  governed  not  only  by  abilities  but  also  by  predilections  or  tendencies.  This 
additional  aspect  of  critical  thinking  has  been  termed  dispositions  by  a number  of  educational  researchers  and 
philosophers  (Baron,  1985;  Bereiter,  1995;  Cacioppo  & Petty,  1982;  Ennis,  1986).  As  Norris  (1985)  points  out, 
“One  must  have  the  disposition  to  think  productively  and  critically  about  issues,  or  else  no  amount  of  skill  in  doing 
so  will  be  helpful”  (p.  40). 

A number  of  taxonomies  of  good  thinking  dispositions  have  been  developed  (e.g.,  Ennis,  1986;  Facione, 
Facione,  & Giancarlo,  1996).  They  all  go  beyond  analyzing  ability  to  perform  cognitive  tasks.  Their  emphasis  is  on 
attitude  and  awareness.  For  instance,  the  dispositions  that  Taube  (1997)  identifies  as  characteristic  of  effective 
critical  thinkers  are  open-mindedness,  cognitive  complexity,  need  for  cognition,  tolerance  of  ambiguity,  and 
reflectiveness.  Perkins,  Jay,  and  Tishman  (1993)  describe  seven  dispositions  they  view  as  central  to  good  thinking. 
Those  are  the  dispositions:  to  be  broad  and  adventurous,  toward  sustained  intellectual  curiosity,  to  clarify  and  seek 
understanding,  to  be  intellectually  careful,  to  seek  and  evaluate  reasons,  and  to  be  metacognitive. 

As  Perkins  and  Tishman  (1998)  point  out,  taxonomies  of  thinking  dispositions  are  normative.  These 
taxonomies  reflect  the  cultural  orientation  of  the  authors  and  their  conceptions  of  how  best  to  gain  knowledge.  In 
many  cases,  the  focus  of  these  efforts  is  on  learners  and  how  to  nurture  the  development  of  critical  thinking 
dispositions  in  learners.  That  is  also  the  focus  of  the  present  paper.  The  purpose  of  this  paper  is  to  explain  the 
importance  of  dispositions  as  an  essential  aspect  of  critical  thinking  and  to  suggest  how  instruction  can  be  designed 
to  promote  learners’  development  of  the  dispositional  side  of  critical  thinking. 

The  Dispositional  Bottleneck:  Sensitivity  to  Occasion 

Most  authors  treat  dispositions  simply  as  tendencies,  for  example  the  tendency  to  consider 
counterarguments  to  one’s  own  position  on  an  issue.  However,  Perkins,  Jay,  and  Tishman  (1993)  have  demonstrated 
that  a full  account  of  intellectual  behavior  requires  three  components:  sensitivity,  inclination,  and  ability.  Sensitivity 
concerns  awareness  of  occasion;  inclination  concerns  motivation  or  leaning;  ability  concerns  capability  to  follow 
through  appropriately.  For  example,  to  attend  seriously  to  the  other  side  of  an  argument  in  naturalistic 
circumstances,  a person  would  need  to  be  sensitive  to  the  occasion  to  seek  otherside  reasons,  inclined  to  invest 
mental  effort  in  examining  the  other  side  of  the  issue,  and  have  the  basic  ability  to  do  so. 

Perkins  and  Tishman  (1998)  have  conducted  studies  to  explore  the  validity  of  their  three -component  model. 
Eighth  graders  read  short  stories  which  contained  thinking  shortfalls.  A series  of  three  tasks  were  used  to  measure 
the  relative  primacy  of  sensitivity,  inclination,  and  ability  as  determinants  of  critical  thinking.  In  the  first  task,  which 
was  designed  to  reveal  sensitivity,  subjects  were  asked  to  underline  parts  of  the  story  they  thought  reflected  poor 
thinking.  The  second  task  presented  the  shortcoming  and  then  asked  the  subjects  whether  they  thought  it  was 
problematic,  and,  if  so,  what  should  be  done  about  it.  This  second  task  was  meant  to  determine  inclination  to  think 
through  the  potential  problem.  In  the  third  task,  after  being  informed  that  there  was  a problem,  the  subjects  were 
asked  to  list  several  options  for  dealing  with  it.  This  final  task  focussed  on  the  subjects’  ability  to  generate 
alternative  options,  independent  of  their  sensitivity  or  inclination  to  do  so.  Results  indicated  significantly  greater 
performance  on  the  ability  task  compared  with  the  inclination  task,  and  significantly  greater  performance  on  the 
inclination  task  compared  with  the  sensitivity  task.  This  result  uncovered  a construct  that  Perkins  and  Tishman 
termed  the  “disposition  effect”:  A substantial  gap  exists  between  what  people  can  do  (ability)  and  what  they  do 


we  wish  to  approach.  Likewise,  we  are  likely  to  notice  those  objects  toward  which  we  have  a strongly  associated 
negative  evaluation — those  that  can  hurt  us — those  that  we  wish  to  avoid.  What  we  “see”  appears  to  be  influenced 
by  accessible  attitudes.  By  orienting  our  attention  to  objects  that  have  the  potential  for  hedonic  consequences, 
accessible  attitudes  ready  us  to  respond  appropriately.  (Fazio,  1995,  p.  265) 

Teachers  and  instructional  designers  can  promote  learners*  sensitivity  to  occasions  that  call  for  critical 
thinking.  They  can  do  this  by  providing  sufficient  opportunity  for  learners  to  engage  in  motivating  and  deliberative 
activities  that  entail  identifying  occasions  for  critical  thinking.  Learners  may  thereby  develop  good  thinking 
dispositions  that  are  strong  and  accessible.  Through  appropriate  activities,  these  dispositions  may  become  the 
default  attitudes  available  during  spontaneous  processing.  In  other  words,  students  might  learn  to  detect  occasions 
for  critical  thinking  as  a natural  habit  of  mind. 

Attitude  Strength  and  Ill-Structured  Problems 

Development  of  strong  attitudes,  attitudes  that  are  readily  activated  from  memory  when  an  individual 
encounters  a situation  associated  with  the  attitude,  depends  upon  a number  of  factors.  Researchers  in  a wide  range  of 
investigations  have  identified  a variety  of  determinants  of  attitude  strength.  In  a review  of  this  research,  Fazio 
(1995)  suggests  that  five  factors  are  major  determinants  of  attitude  strength  and  accessibility  from  memory:  (1) 
direct  experience,  (2)  sensory  experience,  (3)  emotional  reactions,  (4)  freely  chosen  behavior,  and  (5)  attitude 
rehearsal. 

Consideration  of  thinking  dispositions  as  attitudes  that  can  be  activated  in  certain  situations  suggests  that  to 
facilitate  the  learning  of  sensitivity  to  critical  thinking  occasions,  instruction  should  incorporate  into  motivating 
problem-solving  activities  the  attitude-strengthening  factors  Fazio  identified.  Solving  ill-structured  problems  can 
furnish  learners  with  activities  that  meet  these  conditions.  Ill-structured  problems  are  “the  kinds  of  problems  that  are 
encountered  in  everyday  practice”  (Jonassen,  1997,  p.  68).  They  are  problems  situated  in  the  real  world.  Jonassen 
characterizes  ill-structured  problem  solving  as  a design  process  in  which  “the  problem  solvers  must  frame  the  design 
problem,  recognize  the  divergent  perspectives,  collect  evidence  to  support  or  reject  the  alternative  proposals  and 
ultimately  synthesize  their  own  understanding  of  the  situation”  (p.  79).  These  problem-solving  activities  incorporate 
the  above-mentioned  five  attitude-strengthening  factors.  They  afford  learners  opportunities  to  (1)  directly  experience 
occasions  for  critical  thinking,  (2)  engage  a number  of  senses  in  the  richness  of  real-world  problem  situations,  (3) 
react  emotionally  to  those  situations,  (4)  freely  choose  a path  to  solution,  and  (5)  engage  in  a series  of  activities  that 
provide  repeated  opportunities  to  rehearse  critical  thinking  dispositions. 

Direct  experience  in  opportunities  for  critical  thinking 

Attitudes  based  on  direct  experience  are  more  likely  to  be  activated  and  influence  subsequent  behavior  than 
those  based  on  indirect  experience  (Fazio  & Zanna,  1981).  In  a direct  experience,  the  individual  interacts  with  an 
attitude  object  or  commits  himself  or  herself  to  an  issue  or  event.  This  contrasts  with  an  indirect  experience,  in 
which  the  person  forms  an  attitude  on  the  basis  of  nonbehavioral  information  such  as  reading  about  the  attitude 
object.  Solving  ill-defined  problems  through  authentic  activities  is  one  way  to  involve  learners  in  direct  experiences 
that  incorporate  situations  that  call  for  critical  thinking.  Brown,  Collins,  and  Duguid  (1989)  describe  authentic 
activities  as  “the  ordinary  practices  of  the  culture’*  (p.  34).  A problem  solver  involved  in  authentic  activities 
addresses  a problem  within  the  framework  of  the  context  that  created  it.  Authentic  activities  provide  in-context 
learning  similar  to  that  available  to  people  participating  in  craft  apprenticeships.  Solving  ill-structured  problems 
through  authentic  activities  can  facilitate  the  learning  of  sensitivity  to  critical  thinking  occasions  by  providing 
situations  in  which  the  learner  directly  experiences  the  context  of  a problem.  By  committing  to  critical  thinking 
activities  in  such  situations,  a learner  can  form  strong  associations  between  the  situations  and  the  need  to  think 
critically  about  problems  manifested  in  those  situations. 

The  role  of  sensory  information 

Incorporating  a rich  variety  of  sensory  experiences  into  problem-solving  situations  may  also  strengthen 
critical  thinking  dispositions.  Wu  and  Shaffer  (1987)  provide  evidence  that  attitudes  based  on  sensory  reactions  may 
be  more  accessible  from  memory  than  attitudes  formed  less  directly.  Sensory  experience  of  a situation  contributes  to 
formation  of  strong  attitudes  towards  the  situation,  attitudes  which  are  then  more  likely  to  influence  decision  making 
in  similar  situations.  Again,  authentic  tasks  that  include  the  context  for  ill-structured  problem  solving  can  provide 
learners  with  advantageous  conditions.  Visual,  auditory,  olfactory,  tactile,  and  kinesthetic  stimuli  may  all  be  part  of 
the  “peripheral  features”  of  authentic  tasks.  Brown,  Collins,  and  Duguid  argue  for  the  importance  of  peripheral 
features: 

These  features  are  often  dismissed  as  “noise”  from  which  salient  features  can  be  abstracted  for  the  purpose 
of  teaching.  But  the  context  of  activity  is  an  extraordinarily  complex  network  from  which  practitioners  draw 
essential  support.  The  source  of  such  support  is  often  only  tacitly  recognized  by  practitioners,  or  even  by  teachers  or 
designers  of  simulations.  (1989,  p.  34) 
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Granted,  school  tasks  are  by  nature  to  some  extent  inauthentic  because  students  are  not  practicing  members 
of  communities  within  which  school  tasks  are  embedded  (e.g.,  Lave  & Wenger,  1991).  For  example,  in  one  of  the 
curricular  innovations  reviewed  in  this  paper,  students  are  expected  to  take  on  the  role  of  urban  planners.  While  the 
unit  attempts  to  be  as  faithful  to  the  real  activity  of  urban  planning  as  practicable,  it  cannot  truly  capture  the 
complexity  and  structures  of  discourse  that  professional  urban  planners  engage  in.  Nevertheless,  technology  has  the 
potential  for  increasing  the  authenticity  of  experience  by  providing  sensory  information  (in  the  form  of  video,  audio, 
and  kinesthetic  experiences)  reflective  of,  or  even  recapitulating,  real  phenomena. 

The  role  of  emotions 

The  emotional  reactions  that  a learner  has  to  a task  situation  can  greatly  influence  the  learner’s  attitude 
toward  that  task  and  its  context.  Stronger  object-evaluation  associations  are  formed  when  the  emotional  reaction  to 
an  attitude  object,  such  as  a task  situation,  is  stronger.  More  so  than  other  sources  of  information,  people  trust  their 
emotional  reactions  as  indicative  of  their  attitudes  (Fazio,  1995).  From  a number  of  studies,  Schwarz  and  Clore 
(1988)  provide  evidence  that  individuals  use  their  affective  reactions  as  relevant  information  when  making  evaluative 
judgements.  The  emotional  components  of  memory  appear  to  play  an  important  part  in  an  individual’s  response  to 
current  emotion-inducing  situations.  The  second  author  of  this  paper  has  remarked  earlier  on  the  significance  of 
emotional  memory  to  a person’s  evaluation  of  the  value  of  involvement  in  a current  situation: 

When  a student  faces  an  opportunity  to  engage  in  an  activity,  he  or  she  may  experience  an  emotion 
previously  induced  by  a similar  activity.  Since  her  feelings  index  or  signify  a plethora  of  eliciting  conditions  from 
past  experience,  the  emotion  serves  as  a means  for  unpacking  memory,  revealing  potentially  important  information 
about  the  value  of  participation.  (Middleton  & Toluk,  in  press) 

Sinnott  (1989)  used  think-aloud  protocols  to  describe  and  model  the  processes  involved  in  solving  everyday 
ill-structured  problems.  She  found  emotional  memory  to  be  an  important  component  of  these  processes.  Emotions 
were  often  the  impetus  for  determining  what  the  nature  of  the  problem  was  as  well  as  for  generating  and  choosing 
solutions.  From  its  importance  in  relating  past  experience  to  current  situations,  the  emotional  reactions  of  learners  to 
a situation  can  have  a strong  impact  on  forming  and  strengthening  dispositions  towards  critical  thinking. 

Free  choice 

People  view  their  own  freely  chosen  behavior  to  be  a highly  reliable  and  relevant  indication  of  their 
attitudes  or  dispositions.  One  may  come  to  rely  on  one’s  own  behavior  as  an  attitude  indicator  in  the  same  way  that 
observers  would  rely  on  how  they  see  an  individual  act  to  infer  the  dispositions  that  the  observed  individual  might 
possess  (Bern,  1972).  Behavior  that  an  individual  willing  chooses,  that  thus  allows  for  self-perception,  has  been 
shown  to  promote  formation  of  strong  attitudes  (Fazio,  Herr,  & Olney,  1984).  Since  ill-structured  problems  typically 
do  not  have  a single  solution,  an  important  component  of  the  process  of  solving  ill-structured  problems  is  to  assess 
the  viability  of  alternative  solutions.  To  do  this,  a learner  needs  to  construct  an  argument  for  a preferred  solution  or 
against  alternative  solutions.  In  order  to  support  or  reject  different  perspectives,  the  learner  must  gather  evidence. 

He  or  she  eventually  develops  a personal  position  about  a preferred  solution  (Jonassen,  1997).  Constructing 
arguments  and  articulating  personal  beliefs  during  this  solution  process  engages  the  learner  in  numerous  situations 
during  which  choices  are  freely  made.  These  activities  thus  provide  occasions  for  forming  and  strengthening  critical 
thinking  dispositions. 

Opportunity  for  rehearsal 

Any  object,  issue,  or  event  that  calls  an  individual’s  attention  to  the  evaluation  he  or  she  has  formed  towards 
that  class  of  objects,  issues,  or  events  serves  as  an  additional  instance  of  associative  learning  and  strengthens  the 
individual’s  attitude  (Fazio,  1995).  Repeated  expression  of  an  attitude  in  this  manner  has  been  shown  to  enhance  the 
accessibility  of  the  attitude  from  memory  and  thus  increases  the  ease,  speed,  and  quality  of  decision-making  (Fazio, 
Blascovich,  & Driscoll,  1992).  As  described  earlier,  in  the  process  of  solving  ill-structured  problems  learners  have 
to  weigh  possible  solutions.  This  involves  them  in  constructing  arguments  and  considering  alternatives.  Before 
reaching  a solution,  they  often  proceed  through  iterations  of  restricting  the  alternatives  and  refining  their  arguments 
(Jonassen,  1997).  A learner  attempting  to  solve  this  kind  of  problem  thus  engages  in  a series  of  actions  that  provide 
successive  opportunities  with  similar  attitude  objects  for  rehearsal  of  critical  thinking  dispositions. 

A comparison  of  sorts... 

A variety  of  instructional  programs  include  ill-structured  problem  solving  as  the  approach  taken  for 
instructional  design  and  implementation.  We  focus  on  two  of  these  programs:  The  Jasper  Woodbury  Problem 
Solving  Series  (Cognition  and  Technology  Group  at  Vanderbilt,  1992a),  and  Decision  Making,  a unit  on  problem 
solving  through  algebra  (Middleton  & Roodhardt,  1997).  The  design  of  these  programs  affords  learners  with 
opportunities  to  engage  in  activities  that  can  be  related  to  Fazio’s  attitude-strengthening  factors.  In  addition, 
published  descriptions  of  implementations  of  these  programs  provide  examples  of  positive  changes  in  learners’ 
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new  home  buyers,  and  therefore  wants  single-family  houses.  As  the  two  parties  negotiate,  students  are  asked  to 
develop  feasible  plans  for  making  decisions  on  just  how  the  land  will  be  zoned  and  to  make  recommendations  as  to 
how  these  plans  can  be  carried  out.  The  major  mathematical  content  of  the  unit  centers  on  the  graphical 
representation  of  linear  inequalities — the  basis  for  linear  programming.  However,  the  abstraction  of  the  graphical 
information  into  systems  of  algebraic  inequalities  was  not  emphasized  in  the  unit.  Rather,  the  concepts  of  “fair 
exchange  and  ‘trade-off  were  used  to  help  students  understand  the  linear  relationship  between  x and  y values. 

Students  were  encouraged  to  take  the  position  of  one  of  the  parties  and  argue  their  position  with  their 
classmates.  By  the  end  of  the  unit,  students  develop  a plan  for  rezoning  the  land  given  certain  city  regulations. 

Decision  Making  involves  students  in  the  application  of  math  to  difficult  real  world  problems  with  multiple 
solutions.  Through  the  process  of  designing  and  developing  their  zoning  plans,  as  well  as  defending  their 
recommendations,  the  students  are  clearly  engaging  in  ill-structured  problem  solving.  A number  of  components  of 
the  unit  appear  to  promote  sensitivity  to  occasions  for  critical  thinking.  Table  2 describes  some  of  these  components 
and  the  attitude-strengthening  factors  they  incorporate. 

Opportunities  for  deliberative  processing  occur  at  many  points  in  Decision  Making.  Through  engaging  in 
the  real  world  problem  solving  in  their  roles  as  urban  planners  making  fair-exchange  decisions,  students  repeatedly 
encounter  occasions  that  require  critical  thinking. 

The  motivation  necessary  to  bring  about  and  maintain  deliberative  processing  is  also  evidenced  in  the 
activities  of  students  using  this  unit.  Middleton  and  Roodhardt  (1997)  studied  the  implementation  of  Decision 
Making  by  17  middle  school  teachers  and  their  students  in  four  Midwestern  school  districts.  They  observed  that 
students  were  willing  to  work  beyond  their  teachers’  expectations  for  the  sake  of  creating  quality  projects.  In  most  of 
the  classes,  the  students  were  not  allowed  class  time  to  construct  their  choice  of  final  projects — models  of  their 
rezoning  plans.  Yet,  they  met  after  schooled  on  weekends  to  develop  high  quality,  mathematically  correct  scale 
models. 

When  asked  what  they  thought  of  the  unit,  most  students  agreed  that  it  was  hard,  but  not  too  hard  to  do. 

They  felt  it  was  interesting  because  it  dealt  with  real-life  issues  that  they  had  heard  about  before.  They  understood 
the  application  of  the  context  to  the  fields  of  architecture  and  urban  design,  and  several  students  expressed  a desire  to 
learn  more  about  these  professions.  The  students  also  expressed  a desire  for  more  mathematics  of  this  type. 

Although  the  novelty  of  the  situation  certainly  added  to  their  enjoyment,  the  sustained  effort  students 
demonstrated  over  four  to  six  weeks  of  work  would  suggest  that  the  unit  structure  and  content  tapped  into  the 
structure  of  students’  intrinsic  motivations.  That  is,  the  unit  provided  them  with  a level  of  stimulation  by  the  tasks 
and  a level  of  control  over  those  tasks  that  kept  them  intrinsically  engaged  (Middleton  & Toluk,  in  press). 

The  authentic  tasks  provided  by  Decision  Making  appear  to  be  effective  in  developing  critical  thinking. 
Examples  of  near  transfer  were  reported  by  the  observers.  When  students  were  questioned  with  regard  to  their 
problem-solving  strategies,  they  often  referred  back  to  the  context  of  the  rezoning  conflict  and  said,  “Oh,  it’s  just 
like...,’’  and  used  their  previous  strategies  as  analogies  to  communicate  their  present  thinking.  When  the  observers 
asked  the  students  what  the  mathematical  constraints  to  the  systems  of  inequalities  meant,  the  students  used  the 
relationships  in  the  rezoning  context  to  come  up  with  a general  solution.  As  for  the  dispositional  side  of  critical 
thinking,  observations  did  not  disclose  evidence  of  transfer  to  other  contexts.  Although  this  unit  provides  the 
conditions  needed  for  development  of  strong  and  accessible  dispositions,  the  implementation  study  did  not  reveal 
any  instances  supporting  far  transfer. 

Concluding  Remarks 

The  Jasper  Series  and  Decision  Making  are  just  two  of  a number  of  instructional  programs  that  incorporate 
ill-structured  problem  solving  into  their  design.  Through  the  use  of  activities  focussed  on  solving  ill-structured 
problems,  teachers  and  instructional  designers  may  have  a good  shot  at  promoting  positive  changes  in  learners’ 
thinking  dispositions.  With  involvement  in  authentic  activities,  learners  can  more  directly  experience  problems 
within  real-world  contexts.  Appropriate  use  of  technology  can  add  realistic  richness  to  the  context.  Instruction 
incorporating  ill-structured  problem  solving  also  encourages  learners  to  emotionally  invest  and  freely  choose  as  they 
construct  arguments  and  articulate  a personal  position  on  a problem.  In  addition,  this  kind  of  instructional  design 
promotes  repeated  expression  of  critical  thinking  dispositions  through  the  give  and  take  of  considering  possible 
alternatives  and  developing  support  for  arguments. 

From  our  examination  of  these  two  prototypical  instructional  innovations,  we  can  abstract  a number  of 
design  principles  that  pertain  specifically  to  the  nurturance  of  dispositions  for  critical  thinking  in  the  field  of 
mathematics  and  to  the  stimulation  of  subsequent  achievement: 

Development  of  attitude  strength  takes  time.  Just  as  development  of  the  ability  to  think  critically  in  a domain  takes  considerable 
experience  solving  problems  in  that  domain,  so  does  the  development  of  attitude  strength.  Both  of  the  programs  take  two  to 
four  weeks  to  complete,  revisiting  common  mathematical  and  contextual  themes  over  the  course  of  instruction,  as  opposed  to 
traditional  approaches  to  mathematics  instruction,  which  allow  fewer  experiences  with  concepts  and  contexts,  and  less  time  to 


ERiC 


418 


395 


critical  thinking  dispositions.  In  the  following  sections  we  will  describe  these  instructional  programs  in  relation  to 
the  attitude-strengthening  factors  described  above  and  how  those  factors  might  relate  to  changes  in  learners  critical 
thinking  dispositions,  especially  in  terms  of  sensitivity  to  occasions  that  call  for  critical  thinking. 

The  Jasper  Problem  Solving  Series 

The  Jasper  Woodbury  Problem  Solving  Series  (Cognition  and  Technology  Group  at  Vanderbilt,  1992a)  is  a 
set  of  video-based  adventures,  with  each  adventure  presented  as  a 15-  to  20-minute  story  on  videodisc.  At  the  end  of 
each  story,  a challenge  faces  the  main  characters.  Students  are  given  the  task  of  solving  this  challenge  before  they 
are  allowed  to  see  how  the  characters  in  the  story  solved  it. 

“Rescue  at  Boone’s  Meadow”  is  one  of  a pair  of  Jasper  adventures  that  deals  with  issues  of  trip  planning. 
This  story  involves  an  ultralight  airplane  pilot,  one  of  his  students,  and  their  friend,  Jasper  Woodbury.  During  flying 
lessons  the  student  pilot,  Emily,  as  well  as  the  viewers  learn  a lot  about  the  ultralight,  such  as  the  plane’s  fuel 
capacity,  speed,  and  payload  limits.  While  on  a fishing  trip,  Jasper  finds  a wounded  bald  eagle  and  radios  Emily  for 
help.  The  adventure  ends  with  Emily  posing  this  challenge  to  herself:  What  is  the  fastest  way  to  rescue  the  eagle, 
and  how  long  will  that  take?  Students  presented  with  this  problem  can  go  back  and  search  through  the  video,  which 
contains  all  the  data  needed  to  solve  the  problem.  This  challenge  is  a complex  problem  which  requires  the  students 
to  generate  the  kinds  of  subgoals  that  Emily  must  consider  in  order  to  decide  which  is  the  best  way  to  rescue  the 
eagle. 

The  Jasper  Series  provides  learners  with  richly  detailed  ill-structured  problems.  In  describing  their  series, 
the  designers  explained  that  their  goal  “was  to  create  interesting,  realistic  contexts  that  encouraged  the  active 
construction  of  knowledge  by  learners”  (CTGV,  1993,  p.  52).  Some  of  the  components  of  Rescue  at  Boone’s 
Meadow  that  may  promote  sensitivity  to  occasions  for  critical  thinking  are  described  in  Table  1.  These  appear  to 
incorporate  the  attitude-strengthening  factors  that  Fazio  (1995)  described. 

Recall  that  Fazio’s  model  of  the  influence  of  attitude  on  action  identified  both  opportunity  and  motivation 
as  necessary  for  the  deliberative  processing  needed  to  establish  a disposition  towards  critical  thinking.  Students 
using  Rescue  at  Boone’s  Meadow  receive  extensive  opportunity  to  identify  occasions  where  critical  thinking  is 
appropriate.  For  example,  to  have  a good  shot  at  solving  the  story’s  challenge  they  must  generate  a series  of 
subgoals  based  on  the  contextually  embedded  data. 

As  far  as  motivation  is  concerned,  the  Jasper  Series  provides  the  motivating  conditions  that  Jonassen  (1997) 
describes  as  an  inherent  component  of  instruction  that  incorporates  ill-structured  problem  solving: 

Because  they  are  situated  in  everyday  practice,  [ill-structured  problems]  are  much  more  interesting  and 
meaningful  to  learners,  who  are  required  to  define  the  problem  and  determine  what  information  and  skills  are  needed 
to  help  solve  it  (p.  68). 

Evidence  of  the  motivational  effects  of  the  Jasper  programs  comes  from  the  developers’  assessment  of 
student  attitudes  and  the  comments  provided  by  teachers  (CTGV,  1992b).  Fifth-  and  sixth-grade  students  in  nine 
states  were  given  a questionnaire  at  the  beginning  and  the  end  of  the  school  year  during  which  they  used  at  least  three 
Jasper  adventures,  spending  approximately  one  week  on  each  adventure.  Compared  with  students  in  control 
classrooms,  the  Jasper  students  were  less  anxious  toward  mathematics,  more  likely  to  view  mathematics  as  relevant 
to  everyday  life,  more  likely  to  describe  it  as  useful,  and  more  likely  to  appreciate  complex  challenges.  Teacher 
comments  about  the  Jasper  program  included  some  impressions  of  its  motivational  effect  on  students.  Typical  is  one 
teacher’s  comment: 

We  wrote  letters  to  the  new  fifth  graders  coming  up — giving  them  tips  so  it  will  make  their  year  better.  In 
all  of  them,  I was  looking  over  them,  they  wrote  about  Jasper:  “Just  wait  until  you  get  to  Jasper,”  “It’s  not  just  fun, 
you’ll  learn  so  much.”  (CTGV,  1992b,  p.  308) 

Is  the  Jasper  Series  effective  in  making  critical  thinking  dispositions  strong  and  accessible?  The  developers 
of  the  series  (CTGV,  1992b)  give  examples  of  learning  transfer  in  which  students  spontaneously  made  connections 
between  classroom  activities  and  activities  in  other  classes  or  outside  of  school.  Some  project  teachers  reported  that 
several  parents  noticed  when  they  stopped  at  gas  stations  that  their  children  began  to  ask  questions  about  the  fuel 
capacity  and  efficiency  of  their  car.  Teachers  also  described  the  way  students  came  to  label  complex  everyday 
problems  as  “Jasper  problems.”  In  one  example  of  this  kind  of  problem,  students  picked  up  on  how  a substitute 
lunchroom  staff  failed  to  anticipate  the  meals  it  needed  to  prepare.  Though  these  reports  are  anecdotal,  they  indicate 
that  far  transfer  of  these  dispositions  may  occur  in  a variety  of  situations. 

Decision  Making 

“Decision  Making”  is  a mathematics  unit  designed  for  use  in  Grade  7.  The  unit  presents  students  with  a 
dilemma:  Two  parties  with  competing  points  of  view  are  negotiating  on  rezoning  a reclaimed  landfill  to  build  new 
residences  for  the  city  center.  One  party  wants  to  maximize  the  number  of  people  who  can  live  in  the  area,  and  so 
wants  to  build  high-rise  apartment  houses.  The  other  party  wants  to  rezone  the  land  to  accommodate  the  need  for 
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go  in-depth.  In  instructional  design,  familiarity  does  not  breed  contempt.  In  our  cases,  it  appears  to  breed  strong  positive 
attitudes  to  learn. 


Students  need  to  have  a stake  in  their  learning.  Both  innovations  provide  consistent  and  coherent  experiences  that  attempt  to 
enculturate  the  learner  into  the  ways  of  thinking  and  acting,  as  authentically  as  possible  in  a school  mathematics  environment, 
of  actual  communities  of  practice.  In  Rescue  at  Boone's  Meadow,  learners  have  a stake  in  the  outcome  of  the  rescue.  In 
Decision  Making,  students  must  choose  a perspective  on  urban  planning  that  potentially  would  have  real  consequences  for  the 
ultimate  design  of  the  new  development.  In  each  of  the  experiences  students  encounter,  the  perspective  they  have  chosen 
impacts  on  the  learning  activities  they  engage  in. 

Multiple  perspectives  on  problem  solving  are  built  into  the  design.  In  both  innovations,  students  are  able  to  develop  their  own 
ways  of  solving  the  problems,  and  multiple  perspectives  are  fostered.  A visually-oriented  student  can  utilize  graphical  analysis, 
a symbolically-oriented  student  may  use  algebraic  notation  and  operations,  while  a numeric  thinker  might  use  number  patterns 
in  a table.  These  multiple  methods  are  encouraged  and  fostered  as  bringing  unique  information  to  the  endeavor  of  solving  the 
ill -structured  problem.  Diversity  of  modes  of  thought  is  not  an  afterthought,  but  part  of  the  design  of  the  activities. 

While  just  a terse  list,  these  three  principles  run  counter  to  most  current  approaches  to  curriculum  design. 
Considerable  renegotiations  of  the  conditions  of  learning  is  required  for  environments  such  as  these  to  be  integrated 
into  the  everyday  practice  of  education  that  currently  does  not  allow  enough  time,  integration  of  multiple  ways  of 
knowing,  or  personalization  of  activity  to  really  amount  to  much  in  terms  of  critical  thinking  or  dispositions  to  think 
critically. 

A lingering  concern  is  whether  the  critical  thinking  dispositions  that  this  type  of  instruction  appears  to 
promote  will  actually  transfer  to  contexts  beyond  the  context  of  the  instruction  itself.  This  question  addresses  a 
broader  problem  for  the  use  of  authentic  activities.  Situating  problem  solving  within  a specific  context  apparently 
limits  the  ability  of  a learner  to  transfer  what  is  learned  to  solving  problems  in  other  contexts.  As  Bereiter  (1995) 
points  out,  when  learners  are  given  an  analogous  problem — a problem  in  a different  situation  that  can  be  solved  in 
the  same  way — they  typically  do  not  put  to  use  what  they  learned  in  the  first  situation  unless  they  are  prompted  in 
some  way.  This  may  be  true  for  transfer  of  attitudes  in  general  and  critical  thinking  dispositions  in  particular. 

One  avenue  of  future  research,  then,  could  address  aspects  of  the  transfer  of  learned  dispositions.  Perhaps 
critical  thinking  dispositions  are  activated  by  rather  broad  classes  of  attitude-inducing  situations.  If  that  were  the 
case,  dispositions  might  transfer  more  readily  among  contexts  than  some  other  kinds  of  learning  (such  as  the 
conceptual  or  procedural  knowledge  needed  to  solve  a problem). 

In  every  learning  situation  or  other  context  where  critical  thinking  is  important,  a person’s  ability  is  not 
sufficient  to  predict  how  he  or  she  will  act.  Teachers  and  instructional  designers  should  keep  in  mind  that  learners 
are  also  more  or  less  disposed  to  good  thinking.  Our  efforts  need  to  address  this  attitudinal  aspect  of  critical  thinking 
as  well.  Some  years  ago,  Gordon  Allport  expressed  in  simple  terms  the  importance  of  such  attitudes: 

Attitudes  determine  for  each  individual  what  he  will  see  and  hear,  what  he  will  think  and  what  he  will  do. 
(1935,  p.  806) 
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Table  1.  Critical  Thinking  Components  in  “Rescue  at  Boone's  Meadow 


Attitude -Strengthening  Factor 

Components  Promoting  Sensitivity  to  Occasions  for  Critical  Thinking 

Direct  Experience 

Students  become  participants  in  the  story  as  they  assume  Emily’s  role.  To  solve  their 
problem,  they  must  generate  relevant  sub-problems,  argue  their  positions,  and  select 
among  alternatives.  They  do  this  by  working  directly  with  the  same  information 
available  to  the  character  in  the  story. 

Sensory  Experience 

The  data  needed  to  solve  the  problem  are  embedded  in  the  video  as  a natural  part  of  the 
story.  The  sights  and  sounds  contextually  integrated  into  the  stOTy  provide  rich  sensory 
information. 

Emotional  Reactions 

Students  become  emotionally  engaged  by  their  role  in  solving  the  challenge.  Their 
arguments  about  how  best  to  attempt  saving  the  eagle  are  often  emotion- laden. 

Freely  Chosen  Behavior 

After  viewing  the  story,  students  must  find  and  define  the  problems  to  be  solved.  They 
make  choices  among  realistic  alternatives  when  gathering  evidence  and  reaching 
positions  about  the  solution. 

Attitude  Rehearsal 

The  students  need  to  consider  a number  of  alternatives  and  proceed  through  a series  of 
steps  to  solution  of  the  problem.  Opportunities  for  critical  thinking  occur  at  each  of 
these  steps.  They  can  also  work  on  the  second  of  the  pair  of  adventures  to  get  additional 
practice  on  the  core  schema  of  trip  planning. 

Table  2.  Critical  Thinking  Components  in  “Decision  Making 


Attitude-Strengthening  Factor 

Components  Promoting  Sensitivity  to  Occasions  for  Critical  Thinking 

Direct  Experience 

Students  take  on  the  role  of  architects  and  urban  planners.  To  design  and  develop  plans, 
they  must  apply  mathematics  to  a realistic  situation.  In  negotiating  with  the  opposing 
party,  they  explore  the  feasibility  of  different  plans  and  engage  in  trade-offs  to  reach  a 
workable  solution. 

Sensory  Experience 

Students  apply  the  mathematical  concepts  by  graphing  data  and  creating  scale  models  of 
their  housing  proposal.  This  kinesthetic  experience  authentically  reflects  the  work  of 
architects  and  urban  planners. 

Emotional  Reactions 

In  the  process  of  defining  and  presenting  their  position  on  how  the  land  should  be 
zoned,  students  become  stakeholders  with  an  emotional  investment  in  their  plan. 
Emotions  become  heightened  as  students  on  one  side  work  through  the  negotiations 
process  with  the  other  party. 

Freely  Chosen  Behavior 

While  engaged  in  planning,  making  recommendations  for,  and  negotiating  over  the 
zoning  change,  students  frequently  must  make  choices  and  develop  personal  positions. 
Multiple  models  embodying  the  underlying  mathematical  concepts  in  the  unit  allow 
students  to  choose  a preferred  mode  of  representation  for  solving  each  problem  at  hand. 

Attitude  Rehearsal 

During  the  four  weeks  typically  spent  on  the  unit,  students  encounter  many  occasions 
where  critical  thinking  is  desirable.  These  occasions  occur  in  the  context  of  a real  world 
problem  that  motivates  students  to  continue  to  engage  in  applying  good  thinking. 
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Planning  And  Implementing  Web-Based  Instruction:  Tools  For  Decision  Analysis 
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This  paper  discusses  issues  and  factors  involved  in  making  decisions  on  whether  to  use  web-based 

instruction.  Five  levels  of  web-based  instruction  are  discussed , and  eleven  factors  are  identified  to 

consider  before  making  a decision  to  put  your  classes  on-line. 

We  were  talking  with  some  colleagues  the  other  day  when  one  made  a comment  about  what  free  thinkers 
academics  were.  After  all,  the  whole  purpose  of  tenure  was  to  protect  academic  free  speech.  But  on  closer 
examination,  it  seems  that  the  free  thinking  associated  with  the  academy  does  not  carry  over  into  considerations  of 
technology  for  learning  and  instruction.  In  this  arena,  a herd  mentality  seems  a better  descriptor.  Take  for  example 
the  current  rush  in  academia  toward  implementing  Web-Based  Instruction  (WBI).  How  many  of  your  institutions  are 
scurrying  as  fast  as  they  can  to  get  courses  and  content  on  the  Web?  How  many  times  in  the  last  year  have  you 
engaged  in  a discussion  of  educational  technology  with  a colleague  where  the  topic  of  the  Web  has  not  come  up? 
How  many  times  have  you  engaged  in  that  same  discussion  with  an  administrator  where  the  topic  has  not  come  up? 

If  your  organization  is  like  ours  then  the  answer  to  the  first  question  is  very  few,  and  the  answer  to  the  second  is 
almost  none.  We  in  academia  are  not  alone  in  this  rush  to  judgment,  schools  and  corporations  are  equally  engaged  in 
a frenzied  drive  to  the  Web-based  cliff. 

Granted,  the  Web  is,  or  at  least  appears  to  be,  the  most  significant  development  in  educational  technology, 
and  for  that  matter  communications  technology  in  general,  in  our  lifetimes.  Bran  Ferren  (1996),  executive  vice- 
president  of  Disney's  Creative  Technology  and  Imagineering  division  notes  that  significant  increases  in 
communications  bandwidth  throughout  history  have  been  accompanied  by  significant  changes  in  civilization.  The 
most  familiar  of  these  is  the  tie  between  the  development  of  the  printing  press  and  the  renaissance  in  Europe.  Ferren 
holds  that  the  Web,  or  more  generally  the  Internet,  is  the  most  recent  significant  increase  in  communications 
bandwidth.  He  suggests  that  society  as  we  know  it  will  soon  become  society  as  we  have  only  begun  to  imagine  it. 

So  let  us  stipulate  that  we  believe  the  Web  is  the  future  not  of  educational  technology,  but  of  education. 

Still,  that  future  is  not  yet  here.  The  current  rush  to  conduct  education  and  training  on  the  Web  is  at  best  ill- 
advised  and  at  worst  could  create  a backlash  of  the  disillusioned  masses  when  the  Web  in  its  current  state  fails  to 
deliver  the  magic  educational  bullet  everyone  expects.  We  argue  that  the  Web  will  be  incredibly  valuable  for 
education  in  the  near  and  distant  future,  and  can  be  incredibly  valuable  for  education  today  if  used  appropriately. 

Part  of  this  depends  on  the  technical  future  of  the  web,  and  part  of  it  depends  on  how  we  implement  the  technology 
in  education.  While  we  may  not  participate  directly  in  the  Web’s  technical  future,  we  do  have  a respectable  say  in  its 
implementation  in  education.  So  among  the  questions  we  need  to  be  asking  is  how  can  we  use  the  existing  tools 
effectively? 

The  Web  is  not  appropriate  for  all  things  in  all  situations.  Instead  educators  should  examine  their  own 
needs  and  resources,  and  determine  individually  whether  to  employ  Web-based  instruction.  This  paper  presents  a 
tool  for  determining  whether,  and  more  importantly,  to  what  level  use  of  the  Web  is  advised  in  a given  situation.  We 
present  five  levels  of  Web  use  and  a variety  of  factors  that  should  be  considered  to  determine  which  level  is 
appropriate.  We  hope  that  educators  and  administrators  will  pause  in  their  headlong  rush  to  the  Web  and  consider 
these  factors  before  adopting  it  as  a learning  medium.  To  paraphrase  Aristotle  "the  unexamined  course  is  not  worth 
giving." 

Contrary  to  the  belief  of  the  general  populace,  the  Web  is  not  a neat,  orderly,  well-designed  place.  Because 
of  the  way  in  which  the  internet  was  originally  designed,  it  is  out  of  necessity  amorphous,  messy,  and  chaotic.  Thus, 
dividing  it  into  categories  can  be  a frustrating,  counterproductive,  and  often  impossible  task.  Fortunately,  if  we 
consider  only  aspects  of  the  Web  intended  to  be  used  for  education,  or  perhaps  more  accurately  uses  of  the  Web 
intended  to  the  educational,  the  task  becomes  more  manageable.  We  suggest  five  levels  of  use  of  the  Web  common 
in  schools,  colleges,  and  corporations.  These  levels  represent  a continuum  from  basic  occasional  use  to  advanced 
continual  use.  They  are:  informational,  supplemental,  essential,  communal,  and  immersive. 

Level  0:  No  Web  Use 

To  begin  though,  we  offer  a definitional  Level  0.  By  default,  this  level  implies  no  use  of  the  Web  at  all. 

We  believe  that  this  level  of  Web  use,  though  standard  now,  will  eventually  become  as  uncommon  as  level  five. 

This  transition  will  occur  if  for  no  other  reason  than  to  save  paper. 
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Level  1:  Informational  Web  Use 


The  informational  level  of  Web  use  is  the  most  common  and  easiest  to  manage.  Informational  web  use 
consists  solely  of  providing  relatively  stable  information  to  the  student.  Typically  this  information  is  administrative 
in  nature  and  may  not  convey  course  content  directly.  Students  may  access  this  information  from  time  to  time  during 
the  course  for  reference  purposes,  but  would  not  be  expected  to  review  it  on  a frequent  basis.  Level  one  Web  use 
typically  consists  of  the  instructor  placing  items  such  as  the  syllabus,  course  schedules,  and  contact  information  on 
the  Web  for  student  review.  This  sort  of  information  is  easily  created  by  the  instructor  or  an  assistant,  requires  little 
or  no  daily  maintenance,  and  takes  up  minimal  space  and  bandwidth. 

Level  2:  Supplemental  Web  Use 

The  supplemental  level  of  Web  use  is  becoming  more  common,  is  more  useful  than  the  informational  level, 
and  is  only  slightly  more  difficult  to  manage  than  the  informational  level.  The  key  difference  between  level  two  and 
level  one  is  that  the  supplemental  level  actually  provides  course  content  information  for  the  learner.  As  the  name 
suggests  however,  this  information  is  not  critical  to  course;  it  is  intended  as  an  addendum  to  the  core  content.  In  this 
level  the  main  part  of  the  educational  experience  is  provided  in  a classroom  setting.  Students  may  access  this 
information  frequently  if  not  on  a daily  basis.  Level  two  consists  of  the  instructor  placing  course  notes  and  other 
handouts  on  the  Web.  A typical  example  would  be  a PowerPoint  presentation  saved  as  an  HTML  document  and 
placed  on  the  Web  for  students  to  review  later.  Level  two  use  requires  a bit  more  technical  know-how  by  the 
instructor,  daily  or  weekly  maintenance,  and  low  to  moderate  space  and  bandwidth  depending  upon  the  nature  of  the 
material  placed  on-line.  For  example,  a textual  outline  of  course  notes  would  require  very  little  bandwidth  compared 
to  a 30-slide,  graphic-rich  PowerPoint  presentation.  One  point  to  consider  in  using  supplemental  web  use  is  the 
timing  of  placing  the  information  on-line.  One  colleague  found  that  if  they  placed  lecture  notes  on  the  web  before 
class  that  course  attendance  dropped  significantly. 

Level  3:  Essential  Web  Use 

Essential  refers  to  the  fact  that  the  student  cannot  be  a productive  member  of  the  class  without  regular  web 
access  to  the  course.  The  essential  level  of  Web  use  is  still  fairly  uncommon  today.  At  this  level  the  student  obtains 
most,  if  not  all  of  the  course  content  information  from  the  Web.  At  this  level  one  might  think,  for  lack  of  a better 
example,  of  the  Web  replacing  the  textbook  in  of  course.  (We  suggest  that  this  thinking,  while  convenient,  is 
ultimately  counterproductive  to  the  effective  use  of  the  Web  and  education.  It  is  indicative  of  a traditional,  direct 
instruction  approach  to  education  and  may  limit  more  effective  and  creative  uses  of  the  technology.  It  also  suggests 
that  simply  changing  the  media  makes  a significant  impact  on  the  course  content.)  Course  content  may  be  created  by 
the  instructor  or  compiled  by  the  instructor,  but  is  likely  a combination  of  both.  It  requires  the  instructor  to  have 
good  HTML  skills,  good  instructional  design  skills,  good  graphic  design  skills,  good  information  literacy  skills,  and 
ample  course  development  time.  If  the  instructor  does  not  have  these  skills,  the  instructor  needs  access  to  somebody 
who  does  have  these  skills.  Classes  still  meet  face-to-face,  but  students  are  expected  to  use  the  web-based  course 
materials  extensively.  Essential  course  material  made  available  on-line  requires  that  the  students  take  a more 
proactive  approach  to  insuring  their  own  learning. 

Level  4:  Communal  Web  Use 

The  communal  level  of  Web-based  instruction  is  only  just  beginning  to  receive  a widespread  use.  At  this 
level  classes  meet  both  face-to-face  and  on-line.  Course  content  may  be  provided  in  an  on-line  environment  or  in  a 
traditional  classroom  environment.  Ideally,  students  generate  much  of  the  course  content  themselves.  This  level  goes 
beyond  basic  HTML  used  and  requires  the  use  of  other  on-line  tools  such  as  internet  chat,  bulletin  boards,  and  one 
and  two-way  desktop  video.  This  level  requires  both  the  instructor  and  the  students  to  have  good  HTML  skills  as 
well  as  good  technology  skills  in  general.  At  this  writing,  on-line  group  collaboration  tools  are  generally  not  as  user- 
friendly  and  bug  free  as  one  might  hope.  Novice  technology  users  might  not  be  able  to  get  past  the  frustrations  of 
imperfect  tools  to  get  to  meaningful  interaction  about  the  course  content. 

Level  5:  Immersive  Web  Use 

The  immersive  level  is  still  quite  uncommon  today.  While  several  excellent  examples  of  it  do  exist,  most 
organizations  do  not  have  the  infrastructure  needed  to  support  it,  nor  do  most  instructors  have  the  skills  needed  to 
implement  it.  At  this  level  classes  do  not  meet  face-to-face.  All  of  the  course  content  and  course  interactions  occur 
on-line.  Note  that  we  re  not  referring  to  the  more  traditional  idea  of  distance  learning.  Instead,  this  level  should  be 
seen  as  a sophisticated,  constructivistic  virtual  learning  community.  While  it  may  include  some  degree  of  the 
traditional  content  presentation,  student  practice,  feedback,  and  assessment  practices  found  in  traditional  distance 
instruction,  it  is  more  generally  comprised  of  learner-centered  constructivistic  pedagogies  such  as  cognitive 
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apprenticeship  (Brown,  Collins  et  al.,  1989),  and  anchored  instruction  (Bransford,  Sherwood  et  al.,  1990).  At  this 
level  both  instructor  and  students  need  a high  level  of  technological  expertise  and  sophisticated  learning  strategies. 

In  addition  to  a continuum  of  levels  of  web  use,  we've  currently  identified  1 1 factors  that  influence  the 
desirability  of  WBI.  While  other  factors  exist,  this  set  seems  to  play  the  largest  role  in  determining  whether  and 
which  level  of  WBI  to  use.  Table  1 below  identifies  the  factors.  The  remainder  of  this  paper  discusses  each  factor. 

Table  1:  Eleven  Factors  Influencing  Decisions  to  Use  WBI 


1.  Distance 

2. 

Stability  Of  Material 

3. 

Need  For  Multimedia 

4. 

Need  For  Student  Tracking 

5. 

Number  Of  Students 

6. 

Amount  Of  Interaction  Needed 

7. 

Social  Pressure 

8. 

Need  For  Offline  Reference 

9. 

Infrastructure 

10. 

Comfort  Levels 

11. 

Access 

Factor  1:  Distance 

Distance  refers  to  the  geographical  proximity  of  the  instructor  and  students.  We  represent  distance  as  a 
continuum  ranging  from  a group  of  people  close  proximity  to  a population  that  is  disbursed  over  great  geographical 
distances.  Obviously  if  the  instructor  and  students  are  scattered  over  a wide  geographical  area,  and  it  is  difficult  if 
not  impossible  for  them  to  meet  face-to-face,  then  on-line  instruction  is  likely  a good  way  to  connect  the  instructors 
and  students.  If  the  students  and  the  instructor  can  easily  meet  face-to-face  then  the  Web  will  be  of  less  value.  This 
is  primarily  because  of  the  amount  of  information  that  can  be  exchanged,  and  the  interactions  that  can  occur  in  face- 
to-face  settings,  dwarfs  that  which  is  possible  over  the  Web.  However,  distance  should  not  be  the  overriding  factor. 
There  are  situations  and  circumstances  in  which  even  those  instructors  and  students  in  close  proximity  can  benefit 
from  on-line  learning. 

Factor  2:  Stability  Of  Material 

Stability  of  material  ranges  from  highly  stable  content  to  completely  variable.  For  example,  course  content 
on  anatomy  is  fairly  stable:  a patella  is  always  going  to  be  a patella.  However,  course  information  on  web 
programming  is  incredibly  dynamic,  and  can  change  literally  overnight.  If  the  course  material  is  highly  stable,  then 
the  Web  is  probably  not  the  best  candidate  for  the  delivery  mechanism.  Most  students  would  still  rather  have  the 
portability  and  readability  of  a textbook  over  network-dependent,  low-resolution  screen  text.  In  addition,  highly 
variable  material  requires  frequent  and  consistent  updates  by  the  instructor  or  students  in  the  learning  community. 

An  advantage  of  using  the  web  is  that  distribution  of  content,  or  updates  to  content,  is  the  responsibility  of  the  user. 
This  can  save  both  time  in  distribution,  and  money  in  printing,  binding,  and  shipping. 

Factor  3:  Need  for  Multimedia 

Need  for  multimedia  ranges  from  low  to  high.  Granted  at  this  point  the  ability  to  provide  multi-media  at 
acceptable  speeds  depends  on  bandwidth,  and  the  type  of  media  being  used,  the  Web  can  be  an  excellent  tool  for 
presenting  multimedia  content.  It  combines  pictures,  text,  graphics,  sound,  and  motion  video  seamlessly.  In 
addition,  it  allows  students  to  access  this  material  whenever  and  as  often  as  they  need.  Therefore,  if  the  course 
requires  or  can  benefit  from  multimedia,  then  the  Web  is  a good  solution. 

Factor  4:  Need  For  Student  Tracking 

The  need  for  student  tracking  refers  to  the  desirability  and  degree  of  accounting  for  student  interaction  with, 
and  progress  through  the  course.  WBI  like  all  computer-based  training  is  well-suited  for  collecting  data  about 
students  as  they  interact  with  a course.  In  addition  to  such  features  as  on-line  testing,  it  is  also  possible  to  track 
things  such  as  amount  of  time  spent,  content  examined,  content  created,  and  student  paths  through  the  content. 
Depending  on  how  student  tracking  is  done,  one  can  examine  paths  through  not  only  your  web  materials,  but  to 
external  web  materials  as  well.  Examining  this  type  of  data,  together  with  an  analysis  of  students’  mental  states  (i.e. 
using  verbal  reports  as  data,  (Ericsson  and  Simon,  1984))  allows  instructors  to  create  a rich  model  of  the  learners. 
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Factor  5:  Number  Of  Students 


Though  it  may  seem  counter  intuitive,  the  more  students  a course  has,  the  lower  the  level  of  WBI  that  is 
appropriate.  A fully  immersive  WBI  environment  as  described  in  level  5 requires  significantly  more  preparation 
time  and  classroom  management  time  for  the  instructor  than  a traditional  course.  In  our  experience,  a large  part  of 
this  time  comes  from  interacting  with  individual  students.  We  believe  that  a single  instructor  cannot  manage  more 
than  10  to  20  students  in  this  type  of  environment.  For  every  additional  10  to  20  students  the  instructor  should  have 
some  sort  of  teaching  assistance.  On  the  other  hand,  for  courses  with  hundreds  of  students  it  makes  more  sense  to 
distribute  as  much  information  as  possible  on-line.  Therefore,  for  large  numbers  of  students  levels  1,2,  and  3 are 
indicated.  But  levels  four  and  five,  in  which  more  interaction  is  required,  should  be  reserved  for  smaller  classes. 
This  is  often  times  a struggle  as  you  begin  to  think  about  implementing  web-based  instruction.  Administrators  who 
are  pressed  for  classroom  space  and  the  need  to  generate  more  tuition  related  dollars  often  encourage  faculty  to  take 
more  students  in  on-line  environments. 

Factor  6:  Amount  Of  Interaction  Needed 

By  the  same  token,  traditional  didactic  courses  can  benefit  from  levels  1,2,  and  3 of  WBI.  But  courses  in 
which  a more  Socratic  method  is  employed  require  the  interactivity  found  in  levels  four  and  five.  Email,  chat  rooms, 
bulletin  board  systems,  and  desktop  video  can  be  great  facilitators  of  interaction.  Interestingly,  the  skills  which 
allow  some  students  to  participate  more  in  face-to-face  discussions  may  not  be  as  valuable  in  on-line  discussions. 

For  the  type  of  synchronous  interaction  available  in  chat  rooms,  we  found  that  typing  ability  and  speed  has  a 
monumental  influence  on  participation  The  faster  a participant  can  type  the  more  likely  that  person  is  to  make 
contributions  to  the  chat.  The  more  accurately  a person  can  type  influences  the  perceived  credibility  of  the  chat 
contributions. 

Factor  7:  Social  Pressure 

Although  we  feel  it  is  the  least  pedagogically  valid  of  the  factors  influencing  the  desirability  of  WBI,  social 
pressure  is  the  one  that  drives  most  decisions  about  using  WBI  currently.  While  social  pressure  has  little  impact  on 
the  educational  outcomes  of  WBI,  its  role  in  the  operation  of  larger  educational  systems  is  too  significant  to  be 
ignored.  Institutions  and  organizations  that  offer  no  opportunity  for  Web  based  instruction  may  find  themselves 
viewed  as  old-fashioned  and  out  of  date.  This  finding  could  lead  to  a lack  of  credibility  and  loss  of  resources 
however  undeserved  it  might  be.  It  is  important  to  balance  the  pressure  to  use  Web  based  instruction  with 
economically  and  pedagogically  sound  reasons  for  doing  so.  But  it  must  be  a balance  of  sound  reasons,  and  social 
pressure  should  not  be  ignored  merely  because  it  is  educationally  insignificant. 

Many  argue  that  instructors  should  put  their  classes  on-line  regardless.  It  is  an  interesting  problem,  that  we 
think  can  be  explained  in  the  form  of  an  analogy.  Imagine  that  you  are  a professor  of  molecular  biology,  and  your 
department  chair  comes  to  you  and  says,  “I  want  you  to  write  a self  paced  textbook  to  take  the  place  of  your 
introduction  to  molecular  biology.  Now,  I will  pay  you  for  a summer  course  to  do  this,  but  here  is  what  you  will 
need  to  do.  You  will  need  to  research  and  write  all  of  the  text.  You  will  need  to  proof  read  it,  typeset  it,  print  it, 
bind  it,  and  take  care  of  distributing  it  to  your  students.  Oh,  and  by  the  way,  it  is  possible  that  none  of  your  students 
would  have  ever  seen  a book  before.”  Most  of  us  would  politely  pass  on  this  opportunity  to  someone  else,  but  more 
and  more  this  is  what  is  being  asked  of  faculty  members.  The  social  pressure  of  putting  classes  on-line  needs  to  be 
tempered  with  the  reality  of  the  resources  available  to  put  the  course  on-line. 

Factor  8:  Need  For  On-line  Reference 

More  and  more  content  is  available  on  the  web  with  each  passing  day.  From  real  estate  listings  to 
meteorological  information,  good,  sound,  and  reliable  content  is  available  on  the  web.  In  certain  disciplines,  it  may 
be  impossible  to  not  use  web  based  resources.  For  example  in  the  study  meteorology,  one  can  imagine  that 
instructors  would  want  their  students  to  access  weather  related  sites  on  a daily  basis.  In  an  environment  such  as  this, 
much  of  the  content  exists  without  the  instructor  creating  it.  This  is  a compelling  reason  for  students  to  be  on-line. 
For  other  types  of  content  where  on-line  resources  may  not  exist,  it  may  not  be  the  wisest  idea  to  put  the  course  on- 
line. 

Factor  9:  Infrastructure 

The  infrastructure  of  any  organization  cannot  be  overlooked  when  making  a decision  to  use  WBI.  Consider 
for  a moment  that  in  levels  4 and  5 nearly  all  of  the  course  is  delivered  via  the  web.  If  your  web  server  is  not  robust 
enough  to  handle  heavy  traffic,  or  if  your  Information  Systems  department  finds  it  difficult  to  resolve  server 
problems  quickly,  WBI  may  not  be  a viable  option. 
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There  also  needs  to  be  a support  infrastructure  for  faculty  to  help  them  get  materials  on-line  in  the  first 
place.  While  HTML  at  its  most  basic  level  is  mastered  fairly  easily,  more  complicated  interactions  and  media 
presentations  may  require  external  support.  Students  and  faculty  both  may  enjoy  working  from  home  rather  than  on 
campus.  The  organization  needs  to  provide  appropriate  technical  support  to  make  sure  that  technology  failures  play 
as  little  role  as  possible  in  the  success  of  the  class.  Faculty  need  appropriate  hardware  and  software  support  to  work 
with  the  on-line  environment. 

Factor  10:  Comfort  Levels 

Obviously  faculty  who  have  never  seen  the  web  have  no  business  trying  to  conduct  web-based  classes 
without  significant  training  before  they  begin.  Students  who  do  not  understand  either  how  the  web  works,  or  how  to 
manage  information  on  the  web  will  become  frustrated  in  web-based  environments.  The  level  of  comfort  one  has  in 
the  environment  will  dictate  how  much  one  can  get  out  of  the  environment.  Students  struggling  to  understand  the 
nature  of  menus  and  submenus  on  a web  page  will  likely  spend  more  time  learning  how  to  be  a student  in  the 
environment  that  the  content  contained  in  the  environment.  It  is  a matter  of  time,  and  time  cuts  both  ways  here.  We 
need  to  move  quickly  to  respond  to  pressure  to  have  a web  presence,  and  we  need  to  give  ourselves  time  to  make 
sure  it  is  done  well,  and  that  we  are  comfortable  in  doing  it. 

Factor  11:  Access 

Related  closely  to  infrastructure  and  comfort  levels,  access  refers  to  whether  or  not  we  can  get  to  the 
materials.  Instructors  need  to  be  able  to  work  from  the  office  and  at  home.  If  they  travel  a lot,  they  need  to  be  able 
to  work  from  the  road  as  well.  There  needs  to  be  access  to  both  hardware  to  run  the  environment  and  software  to 
develop  and  work  within  the  environment.  Access  refers  not  only  to  internet  access,  but  to  equipment  access  as  well. 
Every  time  we  offer  an  on-line  class,  there  will  be  students  who  enroll  in  the  class  with  some  type  of  access 
problems.  Usually  it  is  a problem  with  a web-browser  version,  or  in  modem  speed.  In  some  cases  it  is  as  dramatic 
as  the  student  needing  to  invest  in  a new  computer.  Students  need  to  be  prepared  to  accept  responsibility  for  access, 
surely.  But  additionally,  instructors  need  to  be  prepared  to  address  access  problems,  and  the  institution  needs  to  be 
prepared  to  help  resolve  them. 

Conclusions 

We  aren't  sure  if  there  are  any  real  conclusions  yet:  what  we  have  to  say  is  based  on  experience,  research, 
and  analysis.  It  is  also  based  on  the  state  of  web  technology  as  of  the  writing  of  this  paper.  At  this  juncture  in 
history,  we  can  foresee  the  five  levels  remaining  fairly  consistent.  While  level  0 may  cease  to  exist,  institutions  take 
on  the  responsibility  of  reaching  level  1,  and  faculty  for  reaching  level  2,  the  higher  levels  require  more  of  a 
conceptual  shift  among  faculty  and  student.  This  requirement  may  make  widespread  use  of  these  levels  of  WBI  a 
longer  time  coming..  The  factors,  however,  are  quite  malleable.  As  technology  changes,  so  will  the  factors. 

Keeping  up  with  the  factors  will  be  difficult  in  traditional  outlets.  While  this  report  was  prepared  in  February  of 
1999,  it  may  not  make  it  to  print  until  July  of  1999.  Four  months  in  web-time  may  be  an  eternity.  We  need  to  be 
prepared  to  not  only  reap  the  rewards  of  web-based  learning,  but  to  manage  the  responsibilities  as  well. 
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Abstract 

The  users  of  content-based  software  programs,  including  hypertexts  and  instructional  multimedia,  rely  on 
the  navigation  functions  provided  by  the  designers  of  those  programs.  Typical  navigation  schemes  use 
abstract  symbols  (arrows)  to  label  basic  navigational  functions  like  moving  forward  or  backward  through 
screen  displays.  In  a previous  study  conducted  by  Boling,  King,  Avers,  Hsu,  Lee  & Frick  (1996)  the  use  of 
concrete  representations  (screen  miniatures)  to  indicate  backward  navigation  functions  was  found  to 
reduce  the  number  of  errors  users  made  in  backward  navigation  significantly.  The  current  study  tested  this 
claim  using  a more  diverse  sample  of  subjects  and  an  electronic  instrument  instead  of  the  original  paper 
instrument.  A hypercard ™ stack  was  developed  in  two  versions,  one  with  screen  miniatures  and  one  with 
arrows  to  represent  certain  backward  navigation  functions,  and  used  to  test  subjects’  ability  to  choose  the 
correct  button  for  a particular  backward  navigation  event.  Subjects  using  the  screen  miniatures  made 
significantly  fewer  errors  than  did  subjects  using  arrows. 

Icons  for  backward  navigation  in  content-based  software 

Graphical  user  interfaces  as  they  are  used  in  most  interactive  content-based  software  rely  on  representations 
within  that  software  to  supply  the  users  with  its  basic  functions.  Representations,  or  signs,  are  fundamental  elements 
of  communication.  Signs  communicate  by  virtue  of  “ ...  a three-way  relation  between  the  representamen  (that  which 
represents),  the  sign's  object  (that  which  is  represented),  and  its  mental  interpretant  (the  situated  intelligence  that 
performs  the  necessary  substitution  of  signifier  for  signified)”  (Mullet  & Sano,1995;  p.  17 1).  In  human  computer 
interface  design,  the  critical  process  of  representation  depends  on  establishing  a clear  relationship  between  a 
representamen  and  its  object  for  the  users.  To  interact  with  an  information  system,  users  need  to  draw  a connection 
between  each  representamen  and  its  corresponding  system  function. 

A common  functional  element  in  content-based  programs  is  the  hyperlink,  or  electronic  connection  between 
one  screen  representation  and  another.  Hyperlinks  are  often  represented  as  text,  and  for  navigation  they  are  often 
represented  as  icons  or  buttons  with  pictorial  symbols  on  them.  Users  have  to  decode  these  signs  (or  representamens) 
correctly  or  else  they  risk  error  in  navigating  the  program,  which  frequently  results  in  frustration  with  the  programs. 
The  degree  to  which  the  visual  signs  in  a program  are  easily  decoded  by  users  of  the  programs  (or  interpretants)  can 
therefore  be  a critical  factor  in  the  successful  use  of  such  programs  (Norman,  1988). 

The  navigation  metaphor 

Hyperlinks  allow  users  to  retrieve  information  or  pursue  a content-related  experience  (taking  a quiz,  for 
example)  by  invoking  various  content  nodes  of  the  system  as  unique  screen  displays.  This  control  by  users  over  the 
sequence  of  screen  displays  is  often  described  as  navigation  through  an  information  space  (Nielsen,  1990). 

The  navigation  metaphor  has  been  criticized  as  inappropriate  because  considerable  differences  exist 
between  information  space  and  physical  space  (Landow,  1990;  Mayes,  Kibby  & Anderson,  1990).  Stanton  and  Baber 
(1994)  point  out  that  physical  space  is  static  so  the  landscape,  or  physical  space,  metaphor  can  not  capture  the 
flexibility  of  hypertext.  Maps  that  have  been  adapted  from  physical  navigation  to  provide  the  hypertext  users  with 
navigational  aids  have  also  been  criticized  because  in  complex  programs  they  can  only  provide  an  incomplete  picture 
of  the  hyperspace  (Stanton,  Taylor  & Tweedie,  1992).  Consequently,  the  users  might  employ  sub-optimal  search 
strategies  while  depending  on  a map  (Kim  & Hirtle,  1995)  or  might  even  get  lost  in  the  program  (Stanton  & Baber, 
1994). 

In  contrast.  Canter,  River  and  Storrs  (1985)  assert  that  there  are  fruitful  and  direct  parallels  between 
navigating  concrete  environments  and  navigating  data.  Kim  and  Hirtle  (1995)  even  propose  that  a spatial  metaphor 
can  be  used  as  a framework  for  explaining  and  designing  tools  analogous  to  navigational  aids  in  physical 
environments  that  alleviate  disorientation  problems  in  hypertext  systems.  Their  view  is  supported  by  the  results  of 
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Edwards  and  Hardman’s  (1993)  experiment  to  determine  how  individuals  cognitively  represent  a database  in  the 
form  of  a hypertext  document.  Edwards  and  Hardman  conclude  that  individuals  appear  to  be  attempting  to  create 
cognitive  representations  of  hypertext  structures  in  the  form  of  a survey-type  map.  McKnight,  Dillon  and  Richardson 
(1993)  have  also  documented  people’s  use  of  easily  recognizable  screen  displays  as  “landmarks”  in  a process  similar 
to  physical  wayfinding  while  they  are  using  hypertext  documents. 

The  navigation  metaphor  is  intuitively  compelling.  Many  readers  of  this  study  will  recognize  themselves  in 
the  user  of  a content-based  program  who  says  or  thinks,  “I  just  saw  it  a couple  screens  ago  — now  how  do  I get  back 
there ?”  Lakoff  and  Johnson’s  (1980)  discussion  of  orientation  metaphors  as  a fundamental  construct  in  the  human 
conceptual  system  suggests  that  we  could  have  predicted  McKnight,  Dillon  and  Richardson’s  (1993)  observations. 
People’s  metaphorical  speech  concerning  printed  texts  reveals  a form  of  ontological  metaphor  at  work;  specifically, 
“A  text  is  a terrain.”  Consider  the  statement,  “I  found  that  article  pretty  slow  going , but  I managed  to  work  my  way 
up  to  the  discussion  section  where  I got  bogged  down  and  had  to  backtrack  several  times  in  order  to  make  my  way 
through  the  author’s  explanation.”  Previously  cited  observations  of  users  interacting  with  hypertexts  further  suggests 
that,  whether  or  not  such  a view  ignores  some  of  the  unique  properties  of  hypertext,  users  are  employing  the 
conceptual  “text  as  a terrain”  metaphor  to  them.  The  authors  of  this  study  will  therefore  use  navigation  as  a way  to 
describe  and  reason  about  the  design  of  pictorial  hyperlinks  in  content-based  software. 

The  trouble  with  navigating  backward 

It  is  clear  that  backtracking  is  important  in  navigating  in  hypertext  environments.  Stanton  & Baber  (1994) 
found  that  the  most  common  form  of  user  action  in  hypertext  systems  is  backtracking  to  previous  nodes;  such 
backtracking  serves  as  an  error  management  activity.  Schroeder  and  Grabowski  (1995)  found  that  users  had 
considerable  orientation  problems  in  a hypertext  system  where  no  explicit  backtracking  function  was  provided. 
Boling  et  al.  (1996)  found  that  users  made  more  navigational  errors  when  they  were  "going  backward,"  or  returning 
to  a previously-visited  node,  than  they  made  in  "going  forward,"  or  choosing  a destination  they  had  not  visited 
before. 

Seeking  to  explain  the  higher  error  rate  for  backward  navigation,  Boling  et  al.  (1996)  examined  over  400 
navigation  buttons  drawn  from  130  instructional  hypercard  stacks.  They  discovered  that  designers  of  the  hypertext 
products  reviewed  were  highly  consistent  in  their  choice  of  button  representation  for  forward  and  backward 
navigation  involving  only  one  "step,"  or  the  difference  between  linearly  adjacent  screen  displays.  However,  there 
was  a high  degree  of  inconsistency  in  the  designers’  choice  of  button  representation  for  navigating  backward  to  a 
specific  departure  point  or  screen.  What  accounts  for  the  observed  consistency  in  designers’  choice  of  representation 
for  single-step  backtracking  as  well  as  single-step  forward  navigation?  What  accounts  for  the  high  degree  of 
inconsistency  in  designers’  choice  of  representation  for  navigating  backward  to  a specific  departure  point/screen? 

General  vs.  specific  destinations  in  navigation 

Boling  et  al.  postulate  that  designers’  consistency  and  inconsistency  lie  in  the  nature  of  the  two  types  of 
navigation  tasks.  They  argue  that  the  above  two  types  of  navigation,  single-step  navigation  and  specific-destination- 
backward  navigation  tasks  can  be  viewed  as  representative  of  two  types  of  knowledge  --  general  knowledge 
(knowing  that)  and  specific  knowledge  (knowing  that  one)  (Maccia,  1987;  Maccia,  1988).  General  knowledge 
(knowing  that)  is  knowledge  or  knowing  that  is  "not  bound  to  particular  persons,  places,  events,  things  or  their 
interrelationships”  (Frick,  1997b).  In  contrast,  specific  knowledge  (knowing  that  one)  is  knowledge  or  knowing  used 
to  "discern  the  specific  features  that  make  some  entity  unique  - what  sets  it  apart  from  all  else"  (Frick,  1997a,  p. 

113). 

In  the  context  of  navigation,  general  knowledge  suffices  for  tasks  that  involve  moving  by  certain  rules  from 
the  specific  location  or  screen  display  one  is  seeing  now  to  a different  location/screen  display  that  is  unknown  in 
advance.  In  the  physical  wayfinding  analogy,  "knowing  that"  would  be  equivalent  to  arriving  at  a place  after 
following  a navigational  instruction  like  "Go  one  block  down  and  turn  right."  In  hypertext,  an  example  of  "knowing 
that"  would  be  to  "go  from  this  logical  screen  display  to  the  next  one  or  the  previous  one."  In  both  cases,  the  fact  that 
they  have  arrived  at  a certain  location  or  screen  display  is  more  important  than  whether  or  not  they  recognize  the 
particular  location  or  screen  display. 

In  the  context  of  navigation,  specific  knowledge  is  involved  in  such  tasks  of  moving  from  where  one  is  now 
to  another  specific  location  or  screen  display  that  one  has  visited  previously.  In  the  physical  wayfinding  analogy, 
"knowing  that  one"  would  be  equivalent  to  arriving  at  "my  house"  after  following  the  navigational  instruction,  "Go  to 
your  house."  In  the  hypertext  context,  an  example  of  "knowing  that  one"  would  be  to  arrive  at  the  menu  screen  after 
following  the  instruction,  "Go  to  the  menu  screen  for  this  section."  In  both  examples,  what  is  important  to  users  is 
that  they  have  arrived  at  the  specific  place/screen  display  that  was  intended  and  that  they  have  not  arrived  at  any 
other  one. 


Concrete  representations  of  screens  as  landmarks 

Darken  and  Sibert  (1996)  define  landmark  knowledge  as  "static  information  about  the  visual  details  of  a 
specific  location."  As  such,  landmark  knowledge  is  indeed  a form  of  specific  knowledge  in  which  the  wayfinder 
remembers  or  recognizes  a distinct  element  in  the  environment  and  relies  on  that  element  as  a central  location  or 
departure  point  to  further  explore  the  rest  of  the  environment.  We  argue  that  users  of  hypertext  programs  could  rely 
on  the  distinctive  visual  features  of  specific  screen,  in  the  form  of  concrete  pictorial  representations  on  hyperlink 
buttons,  as  a form  of  landmark  wayfinding  for  backward  navigation. 

Landmark  knowledge  may  be  useful  in  hypertext  navigation  if,  and  only  if,  people  can  recognize  that  they 
have  seen  distinctive  visual  features  of  the  specific  screen  displays  involved.  Past  research  in  visual  recognition  has 
provided  ample  and  encouraging  evidence  in  this  regard.  Empirical  studies  show  that  humans  have  remarkable 
recognition  memory  for  visual  images  they  have  previously  seen  (Standing,  1973;  Paivio,  1971).  More  impressively, 
people  are  capable  of  recognizing  visual  images  even  when  these  images  have  undergone  substantial  transformation 
(of  size,  not  orientation)  subsequent  to  the  first  viewing  (Winn,  1993). 

Since  people  do  seem  to  use  landmarks  in  hypertext,  and  they  are  capable  of  recognizing  a seemingly 
limitless  number  of  images  as  ones  they  have  seen  before,  the  authors  speculate  (as  did  Boling,  et  ak,  1996)  that 
miniature  representations  of  screens  might  function  more  effectively  as  representations  of  backward  navigation  to 
specific  locations  than  do  abstract  representations,  like  variations  of  left-facing  arrows  or  text-based  buttons  of  the 
“Return  to  submenu”  type.  The  use  of  miniatures  for  backtracking  is  not  unprecedented  in  hypertext  applications, 
although  the  examples  known  to  the  authors  have  been  “history  list”  type  implementations  (Ayers  & Stasko,  1996; 
Nielsen,  1990;  van  Dam,  1987). 

Such  representations  would  preserve  the  context  and  much  of  the  distinctive  detail  in  the  previously- vie  wed 
screen,  which  would  not  happen  if  only  a portion  of  the  screen  were  represented  on  the  button.  Screen  miniatures 
would  also  obviate  the  necessity  for  designers  to  struggle  with  the  difficult  task  of  representing  “low  imagery”  verbs 
like  “to  return”  (Rogers  & Obome,  1987)  and  allow  them  to  use  images  that  stand  for  nouns  (“that  screen/location”) 
instead  of  verbs  (“go  back”)  (Krull,  1988). 

Concrete  representation  for  navigation  backward  to  specific  destinations 

In  light  of  the  above  evidence,  the  authors  argue  that  the  use  of  specific  imagery  (a  picture  of  the  specific 
location  or  screen  display  where  the  user  wants  to  go)  should  be  more  helpful  to  the  user  than  the  use  of  abstract  or 
general  imagery  (arrows)  in  representing  specific  navigational  functions.  We  expected  that  users  would  make  more 
errors  in  choosing  navigation  buttons  for  specific  backward  navigation  when  the  buttons  contained  abstract  symbols 
(arrows)  than  when  they  contained  concrete  representations  of  destinations  (screen  miniatures). 

Method 

Instrument 

To  test  the  hypothesis,  a mocked-up  hypertext  program  was  developed  by  adapting  a commercial  product, 
Beethoven™.  The  screen  of  the  instrument  was  divided  into  an  upper  part  that  showed  the  actual  screen  from  the 
hypertext  program  and  a lower  part  where  the  subjects  were  presented  a navigation  task  (see  Figure  1). 
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Figure  I.  Screen  from  study  instrument  showing  part  of  the  linear  navigation  path  preceding  an  actual 
navigation 


You  are  looking  for  information  about  the  percussion  instruments. 
Please  click  on  'Percussion*  to  go  to  this  section 


Subjects  were  given  the  same  set  of  28  navigational  tasks.  To  ensure  that  subjects  had  previously  visited  the 
screens  to  which  they  would  later  be  asked  to  navigate  “backward,”  subjects  pursued  a simulated  navigation  path 
through  a series  of  two  to  four  related  screens  before  they  were  presented  with  a real  navigational  task  (see  Figure  2). 


Figure  2.  Screen  from  study  instrument  showing  a backward  navigation  task . 


Alter  reading  inis  section  you  want  to  return  to  the  chapter  menu 
to  choose  another  instrument  family,  the  brass  instruments,  to 
explore. 

Click  on  the  button  mat  will  allow  you  lo  do  th3t 
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The  tasks  provided  the  context  for  a navigational  action,  sometimes  backward  navigation,  and  then 
requested  the  user  to  click  the  button  that  s/he  would  choose  in  order  to  perform  this  action  (see  Figure  2).  For 
example,  a typical  task  would  read: 

After  reading  this  section  you  want  to  return  to  the  chapter  menu  to  choose  another  instrument  family,  the  brass 
instruments,  to  explore.  Click  on  the  button  that  will  allow  you  to  do  that. 

A separator  screen  was  built  in  between  the  tasks.  Subjects  could  click  a button  on  this  screen  to  go  to  the 
next  task  whenever  they  were  ready. 

Although  this  study  concentrated  on  the  errors  made  in  backward  navigation,  the  tasks  involved  buttons  of 
all  major  navigation  functions  to  prevent  the  subjects  from  focussing  only  on  the  backward  navigation  buttons.  The 
start  and  end  time  of  the  experiment,  the  time  spent  on  each  screen,  and  the  names  of  the  buttons  clicked  by  the 
subject  were  recorded  in  a log  file  as  subjects  completed  the  instrument. 

Two  versions  of  the  program  were  developed  to  measure  the  effectiveness  of  concrete  and  abstract 
representation  in  the  use  of  buttons  for  backward  navigation.  Both  versions  provided  seven  typical  hypertext 
functions  represented  by  a set  of  buttons.  These  functions  were  “quit”,  “more  information”,  “help”,  “go  back  to  main 
menu”,  “go  back  to  chapter  menu”,  “previous”,  and  “next.”  The  only  difference  between  the  two  versions  of  the 
instrument  was  that  “the  go  back  to  main  menu”  and  “go  back  to  chapter  menu”  buttons  were  represented  differently. 
In  one  version  these  functions  were  represented  by  abstract  pictorial  representations  (arrows).  In  the  second  version 
they  were  represented  by  miniaturized  images  of  the  screens  to  which  they  were  linked  (see  Figure  3). 

Figure  3.  Comparison  of  all  buttons  appearing  in  two  versions  of  the  study  instrument  with  their  respective 
functions. 


Function  Abstract  representation  Concrete  representation 

Quit 


More  information 


Help 


Go  back  to  Main  Menu 

ii 


Go  back  to  Chapter  Menu  ft 
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ii 

H 

Next  T f 


Subjects  and  Procedure 

Thirty-five  subjects  representing  different  ethnicity,  nationality,  and  educational  background  participated  in 
this  experiment.  The  gender  ratio  of  the  35  subjects  was  63%  female  to  37%  male.  The  majority  of  the  participants 
were  between  20  and  29  years  old  (77%).  Twenty  percent  were  between  30  and  39  years  old.  One  participant  (3%) 
was  between  50  and  59  years  old.  Nearly  a quarter  of  the  participants  (22%)  were  enrolled  in  undergraduate 
programs,  75%  were  enrolled  in  graduate  programs.  Three  percent  (one  participant)  were  not  students.  Half  of  the 
participants  were  majoring  in  varying  fields  in  education  (51%)  while  the  remaining  half  were  distributed  among 
several  other  subject  areas  (see  Table  1). 


Table  I.  Proportions  of  participants  with  different  majors  in  the  sample 


Major 

Business 

Computer  Science/Telecommunications 
Education 

Library  and  Information  Science 

Natural  Sciences 

Other 


Participants  (Number/Percent) 
3 8.6% 

3 8.6% 

18  51.3% 

3 8.6% 

2 5.7% 

6 17.2% 
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All  participants  in  the  study  reported  that  they  had  used  graphic  browsers  before.  Likewise,  all  participants 
except  one  had  experiences  with  application  programs  that  utilized  navigational  tools  similar  to  those  presented  in  a 
short  demographic  survey  that  they  filled  out  after  completing  the  instrument.  Almost  three  fourths  (74.3%)  of  the 
participants  had  no  prior  experience  with  hypercard.  The  remaining  25.7%  who  had  used  hypercard  before  reported 
to  be  either  relatively  inexperienced  (14.3%)  or  moderately  experienced  (1 1.4%).  No  participant  considered  himself 
as  very  experienced  in  using  hypercard.  Thus,  the  participants  could  be  considered  as  having  some  experiences  with 
iconic  representations  for  navigation.  However,  their  familiarity  with  the  default  hypercard  icons  used  in  this  study 
was  fairly  low  for  most  of  the  subjects. 

Subjects  were  randomly  assigned  to  one  of  the  two  treatment  condi tions._Subjects  were  not  aware  of  the 
differences  between  the  two  versions  of  the  program.  They  only  knew  there  were  different  versions  and  they  were 
assigned  to  one  of  them.  A test  session  took  about  20-25  minutes  including  the  time  required  for  introducing  the 
instrument. 

Subjects  were  introduced  to  the  instrument  through  an  opening  screen  on  the  instrument  that  described  the 
tasks  they  would  encounter  in  the  instrument.  They  then  worked  at  the  computers  individually.  The  researchers  did 
not  sit  next  to  the  subjects  so  as  not  to  influence  the  subjects  inadvertently  with  body  gestures  or  verbal  signals. 

Results 


To  ensure  that  the  random  assignment  to  the  two  treatment  groups  (see  the  methodology  section)  yielded  no 
major  differences  between  the  groups,  Pearson’s  chi  square  was  calculated  for  gender,  age,  and  prior  experience  with 
graphical  browsers  and  hypercard.  No  significant  differences  between  the  groups  were  found  for  any  one  of  the 
variables. 

Mean  number  of  errors  and  response  time 

According  to  our  hypothesis,  we  expected  the  participants  in  the  concrete  pictorial  representations  group 
(miniatures  — Group  B)  to  make  fewer  errors  in  backward  navigation  than  the  participants  in  the  abstract  pictorial 
representations  group  (arrows  --  Group  A).  This  expectation  was  confirmed.  The  mean  number  of  errors  in 
identifying  navigation  buttons  for  each  of  the  treatment  groups  is  reported  in  Table  2. 

Table  2.  Mean  number  of  errors  in  identifying  navigation  buttons  for  each  of  the  treatment  groups 


Group 

Mean 

overall 

Standard 

Deviation 

Overall 

Group  A - abstract  pictorial  representations  (arrows) 
N=  18 

10.39 

4.865 

Group  B - concrete  pictorial  representations  (miniatures) 
N=  17 

4.59 

3.743 

A one-tailed  t-test  revealed  a highly  significant  difference  between  Group  A and  Group  B for  the  number  of 
backward  navigation  errors  (t  = -3.937,  p < .000).  For  all  navigation  tasks.  Group  B who  used  concrete  pictorial 
representations  made  fewer  errors  than  Group  A who  saw  abstract  pictorial  representations. 

No  differences  between  the  two  groups  were  found  in  the  mean  response  time  to  the  backward  navigation 

tasks. 

Discussion 

The  results  of  this  study  suggest  that  concrete  pictorial  representations  may  provide  a less  ambiguous  means 
for  backward  navigation  than  do  abstract  representations.  Providing  screen  miniatures  on  navigation  buttons  helped 
the  users  select  the  appropriate  button  for  backward  navigation,  possibly  by  supporting  their  use  of  landmark 
navigation.  In  contrast,  the  group  who  saw  the  abstract  pictorial  representations  made  more  mistakes  relating 
navigational  functions  to  the  different  symbols  represented  on  the  buttons. 

Considerations  for  designing  screen  miniatures 

Special  consideration  needs  to  be  given  to  the  design  of  screen  miniatures  for  navigation  buttons.  In  this 
study,  some  participants  commented  that  it  was  difficult  for  them  to  distinguish  between  the  miniatures  representing 
the  main  menu  and  the  chapter  menu  since  they  looked  similar  at  the  first  glance.  Nielsen  (1990)  made  a similar 
observation.  He  criticized  HyperCard’s  graphical  history  list  that  showed  miniature  pictures  of  the  last  42  screens 
visited,  pointing  out  the  screen  miniatures  could  easily  be  confused  since  they  looked  too  similar.  Thus,  screens 
functioning  as  landmarks  (that  is,  destined  to  appear  as  miniatures  on  buttons)  would  need  to  have  some  unique 
visual  features  that  retain  their  distinctiveness  even  after  undergoing  considerable  reduction  in  size. 
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Miniaturization  of  an  entire  screen  display  results  in  the  loss  of  detail  and,  most  probably,  recognizability. 
One  solution  to  this  design  problem  might  be  to  reproduce  only  a small  portion  of  the  original  screen  display  on  the 
navigation  button.  This  design  approach  was  used  in  Mosaic  G,  a derivative  of  NCSA’s  Mosaic  2.5,  the  popular 
browser  for  the  World  Wide  Web  (Ayers  & Stasko,  1996).  It  may  also  be  observed  in  any  number  of  Web  site 
designs  where  a portion  of  a logo  or  image  map  is  repeated  on  subordinate  site  pages  as  a hot  link  back  to  the  main 
page  of  the  site.  No  empirical  data  has  been  reported  on  this  form  of  navigation  so  far  as  we  are  aware  at  the  time  of 
writing.  From  a gestalt-psychology  perspective,  however,  one  could  anticipate  that  showing  parts  of  the  screen 
without  their  surrounding  context  might  not  be  very  helpful  providing  concrete  “landmark”  signs  (Easterby,  1970). 
Consequently,  in  the  current  study  the  authors  created  miniatures  of  entire  screens,  relying  on  distinctive  elements  of 
the  full  size  screens  to  retain  enough  detail  that  they  were  functionally  recognizable.  Further  study  on  the  possibilities 
and  limitation  of  partial  screen  landmarks  is  warranted. 

Limitations  of  the  study 

The  scenario  of  the  study  was  not  authentic.  In  an  authentic  hypertext  environment,  users  get  feedback  on 
their  navigation  immediately  and  generally  have  the  chance  to  correct  their  errors  right  away.  The  subjects  in  this 
study  were  directed  to  go  through  the  hypertext  stack  in  a predetermined  sequence  and  did  not  receive  feedback  for 
the  errors  they  made.  The  advantages  of  this  approach  were  that  error  rates  for  respective  navigation  buttons  could 
be  recorded  consistently,  and  the  structured  paths  reduced  the  potential  confounding  effects  from  subjects  getting 
lost  in  the  hypertext  program  or  learning  the  program’s  structure. 

Future  Research . 

Research  should  be  conducted  to  study  subjects’  performance  with  screen  miniatures  in  authentic  tasks  and 
authentic  contexts.  Such  studies  might  also  examine  the  speed  and  ease  of  learning  of  the  meanings  of  abstract  versus 
concrete  navigation  buttons. 

Comparison  of  screen  miniatures  versus  detailed  textual  links  may  also  be  warranted.  While  one  might 
expect  faster  recognition  of  individual  concrete  images  (screen  miniatures)  over  recognition  of  textual  links,  the  time 
difference  might  be  offset  by  the  fact  that  the  screen  miniatures  require  users  both  to  recognize  them  and  to 
remember  whether  the  information  they  seek  exists  at  that  previously-visited  location.  Textual  links  might  carry  more 
helpful  semantic  content  that  ultimately  reduces  either  the  time  required  to  make  a choice,  or  errors  in  navigation,  or 
both. 


Summary 

Backward  navigation  to  a screen/location  visited  previously  is  a frequently  used  navigation  function  within 
content-based  software.  The  use  of  screen  miniatures  may  be  a fruitful  design  strategy  for  reducing  error  in  this  type 
of  navigation,  and  might  be  considered  as  an  option  for  backward  navigation  even  though  other  navigational 
functions  within  a single  program  are  represented  differently. 
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Psychological  factors  influencing  World-Wide  Web  navigation 
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Abstract 

This  experimental  study  investigated  psychological  characteristics  and  external  web-site  features 
influencing  learning  and  disorientation  in  web  navigation.  The  research  design  is  a 2X2  ANOVA  with 
mode  of  navigation  (linear,  nonlinear)  and  distracters  (i.e.,  the  presence  of  “seductive”  links)  as  the  two 
factors.  The  paper  reports  findings  of  the  following  dependent  measures  as  they  relate  to  the  two-factor 
research  design:  friendliness  of  the  web  site,  attribution  of  disorientation,  overall  level  of  disorientation, 
confidence  in  navigating  the  web  site  passage,  interest  in  web  site  passage  topic,  accuracy  at  stating  main 
point  of  the  passage,  and  number  of  ideas  recalled  from  content  of  passage.  It  was  found  that  preference 
for  sensation-seeking  behavior  and/or  spatial-holistic  ability  did  not  influence  the  participants'  perceptions 
of  disorientation.  Other  results  indicated  that  recalling  examples  from  the  text  was  facilitated  when  the  site 
had  no  distracters.  But  an  interaction  indicated  that  participants'  conception  of  the  main  point  of  the 
passage  was  negatively  influenced  if  the  site  had  no  distracters  when  in  the  nonlinear  navigation  mode. 

Introduction 

Given  the  rapid  growth  of  the  Internet  and  the  commensurate  need  to  design  improved  systems  for 
participant  access  and  learning,  this  study  investigates  psychological  issues  related  to  navigation  in  the  World  Wide 
Web.  In  this  paper,  I am  going  to  focus  on  results  from  an  experiment  that  relate  to  psychological  characteristics 
(e.g.f  sensation-seeking  behavior,  spatial-holistic  ability,  disorientation,  attribution,  confidence,  perceived 
friendliness  of  web-site  and  interest  in  content),  learning  (e.g.,  accuracy  at  stating  main  point  of  the  passage,  number 
of  ideas  recalled  from  content  of  passage),  and  external  web-site  features  (e.g.,  mode  of  navigation,  the  presence  of 
distracters),  of  World-Wide  Web  navigation. 

These  three  areas  can  be  conceptualized  in  terms  of  a user's  perception  of  disorientation,  defined  as 
uncertainty  in  one's  location.  A common  phenomenon  on  the  World  Wide  Web  is  for  a participant  to  become 
sidetracked  by  other  links  and  become  disoriented  when  seeking  a particular  location.  From  the  participant’s 
perspective,  did  s/he  make  the  right  choice  or  will  s/he  have  to  back-track?  Did  she  temporarily  ignore  her  original 
intent  because  she  has  a high  need  for  novelty  and  stimulation?  Additionally,  much  of  the  inconvenience  of 
disorientation  may  be  perceptual.  Like  waiting  in  line  at  the  grocery  store,  where  some  people  become  livid  while 
others  use  it  as  a chance  to  start  up  a conversation,  disorientation  on  the  web  may  be  considered  as  exciting  or  as  an 
annoying  distraction  that  prevents  the  participant  from  staying  on-task. 

Purpose 

The  purpose  of  this  experimental  study  is  to  evaluate  the  role  of  internal  (psychological  characteristics)  and 
external  (web-site  features)  factors  in  influencing  Internet  navigation  and  disorientation.  This  research  also  facilitates 
the  development  of  web-site  features  that  better  accommodate  navigation  preferences  and  learning.  For  example,  a 
person  low  in  sensation-seeking  tendency  may  prefer  a more  streamlined  approach  to  navigation  with  limited  links 
whereas  a person  high  in  sensation-seeking  tendency  may  prefer  more  varied  navigational  features  with  a greater 
number  of  available  links  or  distracters.  In  terms  of  learning,  the  participants  are  tested  for  content  learned  from  the 
participant’s  navigation  of  the  web  site  passage.  This  has  direct  implications  for  design  and  development  of  web- 
based  learning  environments.  By  assessing  perceived  factors  relating  to  web  navigation,  the  research  provides 
information  regarding  the  web  site  features  that  participants  believe  are  important  for  a web-site's  participant- 
friendliness. 

Participants 

Volunteer  participants  were  recruited  from  Internet  listings  that  advertised  the  study,  drawing  from  a diverse 
nationwide  population.  All  participants  had  prior  experience  navigating  the  Web  as  a requirement  of  participation  in 
the  study.  They  were  compensated  for  participation.  The  seventy-five  participants  represented  a diverse  population, 
with  79%  white  and  21%  non-white  (including  primarily  Asian  and  Hispanic).  The  mean  age  was  30.47  with  a 
standard  deviation  of  1.26.  Of  the  74  participants  reporting  gender,  34  (46%)  were  female  and  40  (54%)  were  male. 
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Experimental  Design 

The  study  was  originally  designed  as  a two-way  ANCOVA  design  with  system  knowledge  as  the  covariate 
because  systems  knowledge  has  been  shown  to  directly  affect  participants’  competencies  at  navigation  (Hill  & 
Hannafin,  1996).  However,  systems  knowledge  scores  were  very  high  (X=4.08  on  five  point  scale)  and  positively 
skewed,  precluding  the  value  of  using  them  as  a covariate. 

Consequently,  the  experimental  design  is  comprised  of  a two-factor  (2x2)  ANOVA  design  with  the  two 
factors  of  navigation  mode  (linear,  nonlinear)  and  distracters  (presence  or  absence).  These  two  factors  relate  to  the 
external  features  of  the  web-site.  There  were  four  web  sites  that  reflect  different  instantiations  of  the  two  factors 
(e.g.,  linear  + distracters;  linear  + no  distracters;  nonlinear  + distracters;  nonlinear  + no  distracters).  The  four  web 
sites  all  contain  nine  web  pages  of  identical  text  that  comprise  a passage  on  Internet  use  for  education , adapted  from 
Andy  Carvin’ s web  site  at  http://edweb.gsn.org/web.intro.html.  The  software  developed  for  the  experiment  randomly 
selected  one  of  four  web  sites  for  each  participant  to  navigate. 

In  the  linear  navigation  condition,  the  participant  is  forced  to  proceed  through  the  web-site  similar  to 
reading  a textbook,  only  able  to  move  forward  and  backwards.  While  this  is  a somewhat  artificial  treatment,  it  was 
designed  to  maximally  contrast  with  the  nonlinear  mode  of  navigation.  In  the  nonlinear  navigation  mode,  the 
participant  has  immediate  access  to  all  pages  at  any  given  time  through  a navigation  bar  on  the  left-hand  column. 

The  distracter  condition  contained  six  distracting  links  placed  throughout  the  nine  pages  of  the  web  site. 

The  six  distracting  links  were  designed  to  figuratively  “seduce”  the  participant  to  click  on  the  link  (e.g.,  David 
Letterman’s  Top  Ten,  jokes,  Dilbert  cartoons)  and  encourage  off-task  behavior.  Each  distracter  was  comprised  of  a 
small  picture  with  a link  indicating  to  “Click  here  to  ...”  The  no-distracter  condition  did  not  contain  any  distracters. 

The  task  for  all  participants  was  to  successfully  find  the  headings  under  which  five  statements  were  located 
within  the  nine-page  online  passage.  Participants  could  access  the  statements  from  any  page  on  the  web  site.  In 
determining  the  appropriate  heading  location,  the  participant  did  not  have  open-ended  access  to  the  Internet;  rather, 
s/he  was  limited  to  the  pre-defined  web  pages  for  each  web  configuration.  The  toolbars  were  de-activated  so  that  s/he 
could  only  access  web  pages  included  in  the  experimental  web  site.  Software  was  developed  in  Perl  to  track  the 
participant’s  path  through  the  web  site  in  terms  of  total  time  spent,  time  spent  on  distracting  links,  and  the  number  of 
distracting  links  selected.  The  whole  procedure  took  approximately  an  hour. 


Research  Procedures 

Following  completion  of  the  initial  task,  the  participant  was  profiled  online  according  to  the  following  two 
psychological  dimensions:  preference  for  sensation-seeking  behavior  and  spatial-holistic  ability.  In  terms  of 
sensation-seeking  preference,  participants  answered  a battery  of  questions  pertaining  to  preference  for  sensation- 
seeking tasks  (fromZuckerman,  1979).  See  Appendix  A for  example  items.  These  34  questions  consisted  of  paired 
statements  from  which  the  participant  would  select  the  one  statement  best  describing  him/her.  For  example:  A)  I 
would  prefer  living  in  an  ideal  society  where  everyone  is  safe,  secure,  and  happy;  or,  B)  I would  have  preferred 
living  in  the  unsettled  days  of  our  history.  The  mean  score  of  the  participants  was  18.50  (sd=  5.83)  with  a possible 
range  of  0-34.  In  terms  of  spatial-holistic  ability,  participants  completed  the  Street  Test  (Street,  1931),  a thirteen-item 
gestalt  completion  test.  This  instrument  assessed  spatial-synthetic  ability  by  requiring  the  participant  to  mentally 
construct  the  whole  picture  from  a partially-represented  figure,  such  as  a figure  of  a bearded  man,  a cat,  or  a 
locomotive.  See  Appendix  B for  two  example  figures.  The  mean  score  of  the  participants  was  7.36  {sd= 2.52)  with  a 
possible  range  of  0-13. 

Additionally,  information  regarding  the  participants’  prior  knowledge  was  collected  in  terms  of  content 
knowledge  of  the  passage’s  topic,  and  systems  knowledge.  Participants’  mean  prior  content  knowledge  of  the  use  of 
the  Internet  for  education  was  3.65  (on  a 1-5  Likert  scale  where  5 is  extremely  knowledgeable)  with  a standard 
deviation  of  1.05.  As  mentioned  previously,  systems  knowledge  scores  were  very  high  (X=4.08  on  five  point  scale) 
and  positively  skewed,  thus  not  lending  much  information  to  differentiate  participants. 

The  following  dependent  measures  were  assessed  through  an  online  questionnaire: 

Psychological  factors 

Friendliness  of  the  system 
Attribution  of  disorientation 
Overall  disorientation 
Confidence  in  navigating  the  site 
Interest  in  passage  topic 


Learning 

Accuracy  at  stating  main  point  of  the  passage 
Number  of  ideas  recalled  from  content  of  passage 
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Predictions 

First,  in  terms  of  psychological  factors,  it  was  predicted  that  individuals  differing  in  sensation-seeking 
preference  and  spatial-holistic  ability  would  prefer  different  web-site  features.  For  example,  a person  low  in 
sensation-seeking  tendency  may  prefer  a streamlined  site  with  limited  links  and  options  whereas  a person  with  high 
sensation-seeking  tendency  may  prefer  more  varied  navigational  features  with  more  links  and/or  distracters. 
Furthermore,  it  was  predicted  that  these  two  factors  would  influence  other  psychologically-related  dependent 
measures,  such  as  perceived  disorientation  and  confidence  with  navigation  on  the  web-site. 

Second,  it  was  predicted  that  different  web  configurations  (as  defined  by  level  of  distracters  and  navigation 
mode)  would  differently  affect  performance  on  learning  and  psychological  measures  (as  listed  1-7  above). 


Results 

Dependent  measures  were  collected  for  the  seven  areas  listed  above,  including  both  psychological  factors 
and  learning.  Results  will  be  presented  in  each  of  these  seven  areas. 

First,  in  terms  of  friendliness  of  the  web-site , participants  were  asked  to  assess  the  friendliness  of  the  site  on 
a Likert  scale  of  1-5.  There  were  no  statistically  significant  results  in  the  two-way  (navigation  mode,  distracters) 
ANOVA  with  site  friendliness  as  the  dependent  measure.  As  would  be  expected  in  considering  all  of  the  participants 
(who  represent  all  four  web  site  combinations),  participants’  confidence  in  navigating  the  site  was  positively 
correlated  to  their  reported  friendliness  of  the  site  (r=.41 1,  p <.0001). 

Second,  in  terms  of  attribution  of  disorientation , participants  were  asked  to  “Consider  the  times  when  you 
felt  disoriented  during  the  task.  Overall,  do  you  attribute  your  feelings  of  disorientation  to  your  self  or  to  the  web 
site?  (with  1 representing  to  the  self  and  5 representing  to  the  web  site.)”  This  question  served  to  assess  to  what 
extent  participants  attributed  (i.e.,  blamed)  navigation  problems  on  the  web  site  as  opposed  to  themselves.  However, 
there  were  no  statistically  significant  results  in  the  two-way  (navigation  mode,  distracters)  ANOVA  with  attribution 
of  disorientation  as  the  dependent  measure. 

Third,  level  of  disorientation  was  determined  from  ten  Likert-scale  questions  from  Beasley  & Waugh’s 
Non-Linear  Media  Disorientation  Assessment  instrument  (Beasley  & Waugh,  1995).  As  expected,  participants’ 
disorientation  scores  correlated  negatively  to  systems  knowledge  (r=-.253,/?<.05),  interest  in  content  (r=-.351, 
/?<.005),  and  confidence  in  navigating  (r=-.605,  p<.0001).  From  a two-factor  ANOVA  (navigation  mode, 
distracters),  there  was  a marginally  significant  main  effect  for  navigation  type  (F=3.318,  p-.Qil ),  where  those  in 
linear  condition  were  more  disoriented  (X=13.58)  than  those  in  nonlinear  condition  (X=10.63).  This  indicates  that 
participants  feel  more  oriented  with  a nonlinear  web-site  format  as  compared  to  a linear  format. 

Fourth,  there  were  no  statistically  significant  results  in  a two-way  ANOVA  where  confidence  in  navigating 
the  experimental  web  site  was  the  dependent  variable.  As  expected,  there  was  a strong  negative  correlation  of 
perceived  disorientation  with  confidence  in  navigating  the  site  (r=-.605,  p<.0001).  Interestingly,  females  reported 
more  confidence  navigating  the  site  than  males  (X=4.68  vs.  X=4.26,  /?=.001),  yet  they  also  reported  more  overall 
disorientation  with  the  site  (X=10.52  vs.  13.24;  p=.07). 

Fifth,  there  were  no  statistically  significant  results  in  a two-way  ANOVA  where  interest  in  passage  content 
was  the  dependent  variable. 

Sixth,  participants  were  asked  to  explain  the  main  point  of  the  passage.  Items  were  scored  on  a 1-5  scale, 
with  a representative  answer  receiving  a “1”  as  “It  provides  a general  discussion  about  the  web.”  and  a representative 
answer  receiving  a “5”  as  “It  analyzes  the  possibilities  the  web  offers  to  educators  and  its  importance  as  a learning 
tool.”  From  a two-factor  ANOVA  (navigation  type,  distracters),  there  was  a statistically  significant  interaction  of 
navigation  type  and  distracters  (F=4.70,  p<.05)  for  the  main  point  score.  This  interaction  indicates  that  participants 
performed  better  in  the  linear  condition  with  no  distracters  (X=3.63  vs.  2.62),  and  in  the  nonlinear  condition  with 
distracters  (X=3.56  vs.  3.42).  Perhaps  the  distracters  in  the  nonlinear  condition  forced  subjects  to  expend  more  effort 
in  discerning  the  meaning  of  the  passage  and  were  thus  beneficial.  In  contrast,  in  the  linear  condition,  the  presence  of 
distracters  negatively  affected  performance  by  not  fitting  in  with  the  linearity  of  the  passage  navigation. 

Seventh,  participants  were  asked  to  list  as  many  benefits  of  using  the  Internet  for  education  as  possible, 
according  to  the  web-site.  They  were  awarded  one  point  for  each  benefit  listed  that  was  stated  in  the  passage.  The 
range  of  scores  was  0-9,  with  the  mean  as  3.34  (sd=1.97).  A two-factor  ANOVA  (navigation  type,  distracters) 
indicated  that  there  was  a significant  main  effect  for  distracters  (F=6.68,  p-. 01),  where  those  in  web  sites  with  no 
distracters  (X=3.90)  performed  better  (listing  more  relevant  benefits)  than  those  with  distracters  (X=2.76).  This 
indicates  that  the  presence  of  distracters  negatively  affects  idea  production  (in  the  form  of  listing  benefits)  following 
the  learning  experience.  Perhaps  the  distracting  links  distracted  the  participant  from  focusing  on  and  then  later 
recalling  passage  content. 
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Discussion  and  implications 

The  first  prediction,  that  preference  for  sensation-seeking  behavior  and  spatial-holistic  ability  factors  would 
influence  perceived  disorientation  and  preferred  navigation  mode,  was  not  supported  based  on  analysis  of  the  data. 
However,  there  was  a marginally  significant  difference  in  main  point  scores  between  those  (N=36)  with  high  spatial- 
holistic  scores  (AT=3.5 1)  and  low  spatial-holistic  scores  (X=3.06,p<.l ),  thereby  indicating  that  spatial-holistic 
ability  facilitated  discerning  the  main  point  although  did  not  interact  with  other  navigational  factors. 

In  terms  of  the  second  prediction,  three  findings  regarding  motivation,  orientation,  and  learning  will  be 
briefly  discussed.  First,  in  terms  of  motivational  factors  regarding  navigation,  there  is  a clear  connection  between 
perceived  web  site  friendliness  and  perceived  orientation  within  the  site.  Second,  in  terms  of  web  site  and  interface 
construction,  this  study  indicates  that  participants  feel  more  oriented  with  a nonlinear  web  site  format  as  compared  to 
a linear  format.  As  would  be  expected,  it  was  found  that  perceived  disorientation  was  highly  negatively  correlated  to 
both  site  friendliness  and  confidence  in  navigation.  Third,  when  factoring  in  learning  from  the  web-site  it  was  found 
that  recalling  examples  from  the  text  (e.g.,  benefits  of  Internet  use  for  education)  was  facilitated  when  the  site  had  no 
distracters.  But  an  interaction  indicated  that  participants’  conception  of  the  main  point  of  the  passage  was  negatively 
influenced  if  the  site  had  no  distracters  when  in  the  nonlinear  navigation  mode.  An  important  caveat  is  that  the 
participants’  task  was  not  to  learn  the  passage;  consequently,  the  two  learning  measures  (stating  the  main  point  of  the 
passage,  and  listing  educational  benefits  of  using  the  Internet)  actually  assess  incidental  learning.  Overall,  this 
evidence  begins  to  suggest  that  participants  may  indeed  learn  more  from  a nonlinear  than  a linear  navigation  mode.  It 
is  not  clear  whether  or  not  the  role  of  distracters,  or  “seductive”  links,  negatively  affects  learning.  Future  research 
should  include  more  comprehensive  learning  measures. 
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Appendix  A : Sample  items  from  sensation-seeking  instrument  (from  Zuckerman,  1979). 

Instructions:  Each  of  the  items  below  contains  two  choices.  Indicate  which  of  the  choices  most  describes  your  likes 
or  the  way  you  feel.  In  some  cases  you  may  find  that  both  choices  describe  your  likes  or  the  way  you  feel.  Please 
choose  the  one  that  better  describes  your  likes  or  feelings.  In  some  cases,  you  may  not  like  either  choice.  In  these 
cases  mark  the  choice  you  dislike  least.  This  scale  should  measure  only  your  likes  and  feelings,  not  how  others  feel 
about  these  things  or  how  one  is  supposed  to  feel.  There  are  no  right  or  wrong  answers.  Be  frank  and  give  your 
honest  appraisal  of  yourself. 


1 

•I  would  like  a job  which  would  require  a lot  of  traveling. 

•I  would  prefer  a job  in  one  location 

2 

*1  am  invigorated  by  a brisk,  cold  day. 

*1  can’t  wait  to  get  indoors  on  a cold  day. 

3 

•I  find  a certain  pleasure  in  routine  kinds  of  work. 

•Although  it  is  sometimes  necessary,  I usually  dislike  routine  kinds  of  work. 

4 

•i  often  wish  1 could  be  a mountain  climber. 

•I  can’t  understand  people  who  risk  their  necks  climbing  mountains. 

Appendix  B:  Sample  figures  from  instrument  assessing  spatial-holistic  ability  (from  Street , 1931). 


A baby 


A bearded  man 
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Introduction 

The  study  employed  a 2 x 2 (Generative  Activities  by  Navigational  Activities)  post-test  only  research 
design  in  examining  learning  while  using  a hypertext  system  in  a linear,  navigational,  linear  generative  or 
navigational  generative  format.  In  Experiment  One,  participants  were  assigned  one  of  the  four  conditions  and 
expected  to  leam  the  information  to  solve  a posed  problem,  while  students  in  Experiment  Two  learned  the 
information  to  pass  a reading  comprehension  test.  Results  from  Experiment  One  revealed  that  students  in 
navigational  condition  outperformed  those  in  linear  condition  with  respect  to  their  problem  solving  score.  While 
there  was  no  significant  difference  between  generative  and  non-generative  conditions,  the  study  found  that  there  was 
an  interaction  term  indicating  that  generative  activity  may  inhibit  navigational  hypertext  system  in  problem  solving. 

In  contrary  to  Experiment  One  results  from  Experiment  Two  revealed  significant  difference  between  generative  and 
non-generative  conditions.  While  no  significant  difference  between  linear  and  navigational  conditions  with  respect  to 
their  reading  comprehension  scores  was  found,  a closer  look  at  the  learners'  reading  comprehension  raw  score  and 
their  dwell  time  revealed  that  learners  assigned  linear  conditions  outperformed  those  assigned  navigational 
conditions  but  this  effect  was  wiped  out  by  their  dwell  time  in  the  final  analysis.  In  both  experiments  the 
computerized  text  used  was  identical. 

Generative  Learning 

In  Wittrock's  (1974,  1978)  generative  model,  he  examined  how  learners  used  background  knowledge  to 
construct  meaning  from  stimuli.  Learners  are  not  passively  but  are  actively  engaged  in  the  construction  of  meaning  as 
it  relates  to  their  beliefs,  experiences,  current  goals,  and  the  context  in  which  learning  occurs.  Linden  and  Wittrock 
(1981)  found  that  students  who  were  instructed  to  generate  associations  for  text  showed  greater  comprehension  and 
fact  retention  than  those  who  were  taught  using  conventional  reading  instruction.  Doctorow,  Wittrock  and  Marks 
(1978)  showed  that  students  who  generated  paragraph  summaries  significantly  increased  their  retention  and 
comprehension  of  the  text  compared  to  control  groups  who  did  not  generate  summaries.  Studies  in  traditional  text 
processing  also  found  that  encouraging  students  to  generate  elaborated  explanatory  answers  about  new  content 
increases  learning  (Pressley  et  al.,  1992).  Generative  and  elaboration  strategies  have  been  shown  to  be  effective 
while  learning  from  text  in  general  (Pressley  & Ghatala,  1990;  Pressley  et  al.,  1992;  Wittrock,  1974,  1978,  1989). 
Their  application  to  hypertext  systems,  where  learners  already  have  many  task  requirements  (i.e.,  navigational 
choices  and  different  goals),  has  not  been  evaluated.  The  results  of  the  study  by  Barab,  Young  & Wang  (in  press) 
indicated  that  there  was  a positive  correlation  between  generative  activities  produced  and  the  reading  comprehension 
scores.  The  design  of  their  study  (Barab  et  al,  in  press)  did  not  separate  the  navigational  capabilities  from  generative 
activities,  because  both  conditions  afforded  navigational  control;  therefore  it  is  still  unclear  if  the  generative  system 
would  differentially  benefit  readers  with  problem  solving  and  reading  comprehension  goals  for  reading. 

In  working  with  generative  hypertext,  the  text  is  being  altered  as  is  the  reader’s  understanding  of  the 
information  contained  in  the  text.  This  has  much  in  common  with  research  in  ecological  psychology,  defining 
cognition  as  a process  of  a dynamic  interaction  in  which  neither  the  environment  nor  the  organism  is  the  only 
primary  cause.  The  literature  on  situated  cognition  contends  that  meaning  is  constructed  as  an  ongoing  interaction 
between  a perceiving/acting  agent  and  an  information-rich  environment  (Clancey  & Roschelle,  1993;  Dent,  1990; 
Gibson,  1986;  Greeno,  1989;  Young,  1993;  Young  & McNeese,  1995).  Organisms  and  their  environments  define 
and  shape  each  other  in  the  process  of  interaction.  Thus  the  learners’  goals  and  intentions  define  and  are  defined  by 
the  situation  in  which  the  information  is  learned.  As  such,  the  interaction  of  individual's  goals  for  reading  and  the 
situational  constraints  contribute  to  how  the  information  is  perceived  and  learned  (Greeno,  Smith  & Moore,  1993; 
Young,  1993).  One  cannot  exist  without  the  other.  Taking  this  view  leads  to  an  emphasis  of  understanding  as  an 
unfolding  process  in  which  information  is  "picked  up”  (Gibson,  1986)  from  the  text  as  a result  of  the  interaction 
between  each  reader's  current  intentions  and  the  affordances  provided  by  the  current  text.  Learning  is  viewed  as  an 
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active  process  in  which  students  modify  and  reorganize  knowledge  while  they  are  learning,  rather  than  as  a passive 
process  where  learners  are  viewed  as  receptacles  of  knowledge  (Greeno,  1989).  Hypertext  in  generative  format 
affords  students  more  opportunities  to  become  actively  involved  with  information  than  do  non-interactive 
environments.  These  interactive  environments  also  encourage  the  concept  of  "mindful  engagement".  Salomon  et  al. 
(1987,  1991)  discussed  that  although  computer  tools  offer  a partnership  with  the  potential  of  extending  the  user's 
intellectual  performance,  the  degree  to  which  this  potential  is  realized  greatly  depends  on  the  user's  mindful 
engagement.  The  introduction  of  contrived  problems  can  help  focus  or  anchor  learning  in  environments  requiring 
large  amounts  of  learner  control,  by  constraining  navigation  toward  those  aspects  of  the  hypermedia  that  are 
consistent  with  the  posed  problem.  The  potential  benefits  are  mediated  by  learner  goals  for  using  the  hypertext 
(Barab,  Bowdish,  Young  & Owen,  1996;  Barab,  1997). 
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Learner  Control  and  Program  Control 

In  spite  of  the  theoretical  appeal  of  hypertext  learning  environments,  empirical  findings  yield  mixed  results 
with  respect  to  the  learning  benefits  of  learner  navigational  control  over  programmed  linear  control  of  instruction 
(Goforth,  1994;  Kinzie  & Sullivan,  1989;  Niemiec  et  al.,  1996;  Steinberg,  1989).  Although  some  studies  have  found 
increased  benefits  of  navigationally  controlled  instruction  (Gray,  1987;  Hannafin  & Sullivan,  1996;  Kinzie,  Sullivan, 
& Berdel,  1988),  others  have  found  that  students  with  high  degrees  of  learner  control  performed  less  effectively  than 
those  receiving  program  control  (Pollock  & Sullivan,  1990;  Ross  & Rakow,  1981).  When  one  examines  these 
isolated  studies,  there  appear  to  be  results  showing  both  advantages  (Gray,  1987;  Hannafin  & Sullivan,  1996;  Kinzie 
et  al.,  1988)  and  disadvantages  (Pollock  & Sullivan,  1990;  Ross  & Rakow,  1981)  of  learner  navigational  control 
opposed  to  programmed  linear  control.  Lawless  and  Kulikowich  (1996),  examining  hypermedia  navigation  as  well  as 
achievement  and  attitudinal  measures,  found  that  some  students  benefited  from  increased  learner  control  while  others 
appeared  lost,  confused,  or  even  apathetic.  Still  others  have  credited  differences  to  prior  knowledge  finding  that 
increased  learner  control  appears  to  be  useful  for  more  knowledgeable  learners  (Shyu  & Brown,  1995). 

With  the  development  of  multimedia  technology  and  World  Wide  Web,  computer-based  non-linear  text  has 
been  made  multi-dimensional  by  including  not  only  text-based  information  but  also  images,  audio  and  video  clips. 
Reading  hypertext  allows  readers  to  interact  with  the  text,  to  make  decisions  about  both  what  information  to  access 
and  sequencing  of  this  information  (Landow,  1992;  Carver,  Lehrer,  Connell  & Erickson,  1992).  These  features  of 
hypertext  have  drawn  much  attention  from  both  researchers  and  educators  (Barab,  Bowdish,  & Lawless,  1997;  Gall 
& Hannafin,  1994;  Heller,  1990;  Landow,  1992;  Nielson,  1990;  Spiro  & Jehng,  1990)  but  little  attention  has  been 
given  to  the  other  type  of  hypertext  which  is  referred  to  as  generative  hypertext  (Barab  et  al,  in  press;  Young,  Barab, 
& Wang,  1997)  based  on  the  works  of  Wittrock  (1989).  Unlike  reading  traditional  hypertext,  reading  generative 
hypertext  requires  readers  to  adopt  a more  active  role  in  reading  and  become  readers/authors  (Barab  et  al,  in  press  & 
Young  et  al,  1997).  This  type  of  hypertext  format  not  only  allows  learners  to  make  decisions  about  what  information 
to  access,  sequencing  of  information,  and  control  the  pace,  but  also  to  change  the  actual  document  by  engaging  in 
generative  activities  while  reading,  thereby  creating  their  own  unique  text.  This  type  of  open-ended  activity  afforded 
by  the  generative  hypertext  can  encourage  readers  to  be  more  mindfully  engaged.  (Salomon  & Globerson,  1987; 
Salomon,  Perkins  & Globerson  1991).  Although  the  benefits  of  mindful  engagement  in  learning  are  appealing,  little 
empirical  evidence  is  available  for  its  effect  in  the  context  of  computer-based  generative  hypertext. 

Predicted  on  the  research  related  to  the  advantages  of  providing  students  with  a problem  focus  (Anderson  et 
al.,  1994,  Barab,  et  al.,  1996,  CTGV,  1990)  and  advantages  of  instructing  students  to  generate  associations  for  text 
(Linden  & Wittrock,  1981)  we  sought  to  test  this  in  two  separate  experiments:  First,  when  readers  had  a specific 
problem-solving  goal  in  mind  for  reading  and  second  when  their  goal  was  simply  to  comprehend  the  text  in  the 
context  of  preparing  for  a traditional  test  of  reading  comprehension.  We  designed  a generative  form  of  the  Nigeria 
hypertext  which  provides  two  Navigational  activities  (Linear  vs.  Navigational)  and  two  Generative  activities 
(Generative  vs.  Non-generative)  in  this  study.  We  were  specifically  interested  in  comparing  students'  problem 
solving  and  reading  comprehension  abilities  when  information  about  Nigeria  was  presented  as:  1)  Nigeria  Hypertext 
in  linear  format;  2)  Nigeria  Hypertext  in  linear  and  generative  format;  3)  Nigeria  Hypertext  in  navigational  format;  4) 
Nigeria  Hypertext  in  navigational  and  generative  format.  We  hypothesized  that  generative  activity  would  benefit 
reading  comprehension  due  to  their  active  processing  while  reading  and  navigational  activity  would  benefit  problem 
solving.  We  also  hypothesized  that  linear  condition  would  benefit  reading  comprehension  while  generative  activity 
would  benefit  linear  hypertext  system. 

In  Experiment  One,  students  were  assigned  one  of  the  four  conditions  and  expected  to  learn  the  information 
to  solve  a posed  problem.  In  Experiment  Two,  students  were  assigned  one  of  the  four  conditions  with  the  expectation 
of  learning  the  information  to  pass  a reading  comprehension  test.  Dependent  measures  included  a questionnaire 
assessing  students'  problem-solving  success  (Experiment  One)  and  reading  comprehension  (Experiment  Two). 
Efficiency  measure  was  calculated  for  both  reading  comprehension  and  problem  solving  scores  to  control  time  spent 
with  the  hypertext  (dwell  time)  given  previous  findings  demonstrating  significant  correlation’s  between  dwell  time 
on  hypertext  learning  and  outcome  scores.  Items  of  dependent  measures  were  developed  to  address  the  issue  of 
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achievement.  Therefore,  items  were  not  measuring  one  particular  construct.  Specific  research  questions  of  this  study 
are: 


1.  Were  there  any  significant  differences  among  the  students  assigned  the  Navigational  conditions  (linear  vs.  non-linear). 
Generative  conditions  (generative  vs.  non-generative)  with  respect  to  their  problem  solving  scores?  (Experiment  One);  and 

2.  Were  there  any  significant  differences  among  students  assigned  Navigational  conditions  (linear  vs.  navigational)  and  Generative 
conditions  (generative  vs.  non-generative)  with  respect  to  their  reading  comprehension  scores?  (Experiment  Two). 

Experiment  One 

Methods 

Participants 

43  undergraduate  students  were  recruited  from  a Northeast  University.  Before  the  experiment,  each 
participant  completed  a consent  form.  All  participants  received  two  extra  course  points  toward  their  course  grade  for 
their  participation.  Participants  include  6 males  and  37  females.  They  were  randomly  assigned  to  one  of  the  four 
conditions. 

Computerized  Lesson 

The  instructional  lesson  used  in  this  study  was  a social  studies  lesson  about  Nigeria  which  was  modified 
from  its  previous  version  (Barab  et  al,  in  press)  programmed  using  HyperCard™.  The  lesson  was  run  under  four 
conditions  (linear  vs.  navigational,  generative  vs.  non-generative)  on  Macintosh  personal  computers.  This 
computerized  lesson  contains  approximately  70  megabytes  of  information,  including  25  single-spaced  pages  of 
textual  information,  22  digitized  images,  and  5 digitized  video  clips.  All  the  content  is  adapted  from  four  high  school 
textbooks,  two  articles  from  the  Nigerian  consulate,  one  articles  from  the  journal  Face  (1998)  and  current 
information  retrieved  from  the  World  Wide  Web.  All  information  has  been  reviewed  for  accuracy  and  completeness 
by  two  native  Nigerians:  a Nigerian  graduate  student  majoring  in  African  history  and  a Nigerian  undergraduate 
majoring  in  Education.  The  content  of  the  treatment  material  remained  the  same  in  all  four  conditions  in  both 
experiments.  However,  to  establish  a problem  solving  goal  in  experiment  One  participants  viewed  a short  digitized 
video  clip  suggesting  that  there  was  an  unbeatable  virus  currently  plaguing  Nigeria.  Although  the  area  of  Nigeria 
where  the  virus  began  was  unknown,  there  was  patient  data  available  on  six  patients  that  could  be  used  to  determine 
the  origin  of  the  virus  in  hopes  of  locating  the  host  and  developing  an  antibody.  However,  these  files  only  provide 
general  descriptions  of  each  patient's  background  that,  when  matched  with  the  correct  ethnic  group,  could  be  used  to 
predict  the  area  of  Nigeria  where  the  virus  most  likely  originated.  Some  of  the  information  was  related  to  the  patient 
profiles  and  can  be  used  to  predict  the  source  of  the  virus,  while  other  information  was  related  to  other  aspects  of 
Nigeria. 

Problem  Solving  Score 

A problem  solving  score  was  calculated  from  learners'  determination  of  the  ethnic  group  of  all  six  patients 
and  their  inference  that  since  four  of  the  six  patients  were  Hausa,  the  virus  most  likely  began  in  the  north  of  Nigeria 
where  most  Hausa  people  live.  Students  were  given  one  point  for  each  patient  correctly  identified  and  2 points  for 
the  overall  choice  of  North,  where  four  of  the  six  patients  came  from,  with  a total  score  of  8 points. 

Procedure 

All  recruited  participants  were  screened  for  students  with  high  prior  knowledge  of  Nigeria.  Virtually 
nothing  was  known  about  Nigeria  by  all  participants  of  this  study.  The  control  of  prior  knowledge  was  dropped  out 
of  the  analysis.  Treatment  conditions  were  randomly  assigned  to  participants  in  both  experiments  and  they  were  told 
that  the  lesson  was  designed  to  help  instructional  designers  to  develop  effective  educational  software  and  would  take 
about  30  minutes.  All  participants  were  given  a brief  computer-based  tutorial  concerning  the  skills  necessary  to  use 
the  software.  Participants  were  instructed  to  click  on  a particular  button  depending  on  which  condition  they  were 
assigned.  To  pose  a problem  solving  context,  participants  at  this  point  watched  digitized  video  clips  in  which  the 
problem  was  introduced.  While  participants  made  navigational  choices,  the  computer  program  generated  a log  file  to 
maintain  a continuous  record  of  time  they  spent  on  each  node,  their  navigational  path,  their  personal  journal  notes, 
card  headings  generated,  naming  of  geographical  and  ethnic  maps.  After  completing  the  computer-based  section  of 
the  experiment,  students  completed  their  problem-solving  responses  or  a reading  comprehension  questionnaire 
depending  on  which  experiment  they  were  assigned.  The  log  file  data  were  examined  to  determine  the  total  time  each 
participant  spent  studying  the  lesson  and  was  used  to  calculate  efficiency  measure  to  control  for  dwell  time  on  the 
computerized  lesson.  The  efficiency  measures  of  reading  comprehension  and  problem  solving  was  calculated  by 
summing  up  the  number  of  correct  answers  divided  by  the  total  time  and  then  multiplied  by  1000  to  give  whole- 
number  values. 
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Results 


One  participant  was  identified  as  a Native  Nigerian  and  was  dropped  out  of  the  analysis.  The  total  number 
of  participants  in  the  study  was  43.  Randomized  assignment  was  ensured  by  the  design  of  the  study,  so  the 
preliminary  analyses  focused  on  screening  data  to  ensure  the  accuracy  of  data  entry  and  evaluation  of  sample 
distribution,  linearity  and  normality  of  the  scales  used  in  this  study.  No  univariate  outliner  (standardized  score  > 2.5) 
was  found  after  the  evaluation  of  the  statistical  assumptions  but  the  evaluation  revealed  non-normality  of  the 
outcome  variable  (problem  solving  efficiency  score).  Statistical  transformation  suggested  by  Tabachnick  & Fidell 
(1996)  was  performed  to  stabilize  the  variance  of  the  outcome  variable. 

The  results  of  the  Analysis  of  Variance  disclosed  that  there  was  a significant  difference  between  the  two 
navigational  activities  (Linear  vs.  Navigational)  with  the  navigational  activity  demonstrating  superior  problem 
solving  efficiency  score  (F=20.459,  p <.00 1).  Although  there  was  no  significant  difference  between  generative  and 
non-generative  conditions,  there  was  a significant  interaction  among  the  navigational  activities  and  the  generative 
activities  (F=4.124,  p <.05),  which  indicates  that  generative  activities  interfered  learners  when  they  were  given  more 
freedom  to  navigate  to  solve  a problem. 

Experiment  Two 

Methods 

Participants 

44  undergraduate  students  were  recruited  from  a Northeast  Land  Granted  University  in  this  study.  Before 
the  experiment,  each  participant  completed  a consent  form.  All  participants  received  two  extra  course  points  toward 
their  course  grade  for  their  participation.  Participants  include  12  females  and  32  females.  They  were  randomly 
assigned  to  one  of  the  four  conditions. 

Computerized  Lesson 

The  same  instructional  lesson  used  in  this  experiment  as  in  Experiment  One.  However,  instead  of  watching 
the  video  clip  which  was  used  to  introduce  the  problem  in  Experiment  One,  participants  were  instructed  to  learn  the 
information  in  preparation  for  a set  of  reading  comprehension  questions.  All  other  information  in  the  four  conditions 
was  identical. 


Reading  Comprehension  Measure 

The  reading  comprehension  measure  focused  on  factual  information  derived  explicitly  from  the 
computerized  lesson.  The  scale  was  designed  to  measure  retention  and  recall  of  the  factual  information  directly 
related  to  the  social  studies  lesson.  The  Scale  was  composed  of  blank  fill-in  items  with  a total  of  30  blanks.  They 
were  created  from  content  that  spans  all  the  cards  in  the  computerized  social  studies  lesson  and  have  been  reviewed 
by  a content  expert  (a  high-school  history  teacher),  two  educational  psychologists  and  one  doctoral  graduate  student. 
Data  collected  from  the  previous  study  (Barab  et  al,  in  press)  have  been  used  to  perform  item  analysis  of  the  reading 
comprehension  scale.  Items  that  had  a correct  response  rate  above  95%  or  below  5%  were  removed,  so  the  final  pool 
of  items  is  reduced  to  22  items.  The  overall  score  for  the  reading  comprehension  measures  were  calculated  by  adding 
up  the  number  of  correct  responses  (29),  with  a maximum  score  of  29  points. 

Procedure 

Procedures  were  identical  to  Experiment  One,  except  that  after  they  completed  the  program  they  filled  out  a 
reading  comprehension  questionnaire.  The  log  file  data  were  examined  to  determine  the  total  time  each  participant 
spent  studying  the  lesson  and  was  used  to  calculate  efficiency  measure  to  control  for  dwell  time  on  the  computerized 
lesson  as  discussed  in  Experiment  One. 


Results 

One  case  was  dropped  out  of  the  analysis  because  of  incomplete  data  set  due  to  computer  program  crash  in 
the  experiment.  The  total  number  of  participants  in  the  study  was  44.  Randomized  assignment  was  ensured  by  the 
design  of  the  study,  so  the  preliminary  analyses  focused  on  screening  data  to  ensure  the  accuracy  of  data  entry  and 
evaluation  of  sample  distribution,  linearity,  and  normality  of  the  scales  used  in  this  study.  The  evaluation  of  the 
statistical  assumptions  found  1 univariate  outlier  (standardized  score  > 2.5).  It  was  removed  from  the  analysis.  The 
evaluation  also  revealed  non-normality  of  the  outcome  variable  (reading  comprehension  efficiency  score)  statistical 
transformation  suggested  by  Tabachnick  & Fidell  (1996)  was  performed  to  stabilize  the  variance  of  the  outcome 
variable.  The  total  cases  included  in  the  final  analysis  were  43. 

The  results  of  the  Analysis  of  Variance  revealed  that  there  was  a significant  difference  between  the  two 
Generative  activities  (Generative  vs.  Non-generative)  with  generative  activities  help  improving  reading 
comprehension  significantly  (F=5.029,  p <.05).  There  was  no  significant  difference  between  Navigational  activities 
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(Navigational  vs.  Linear)  while  reading  and  no  interaction  term  among  the  two  navigation  activities  and  two 
generative  activities  was  found. 

Discussion 

The  results  of  this  study  suggest  that  increased  levels  of  learner  control  are  beneficial  when  individuals  are 
using  a hypertext  program  to  solve  a specific  problem.  In  this  study,  individuals  who  were  free  to  navigate  directly  to 
those  cards  of  information  they  deemed  appropriate  did  significantly  better  at  the  problem-solving  task  than  those 
who  proceeded  through  the  document  in  a linear  manner.  This  finding  is  consistent  with  the  findings  in  Barab  et  al's 
study  (in  press).  While  there  was  no  significant  difference  between  students  assigned  the  Linear  Non-generative 
Condition  and  Linear  Generative  Condition,  students  assigned  Navigational  Generative  condition  did  not  do  well  on 
problem  solving  comparing  with  those  assigned  Navigational  Non-generative  condition.  This  finding  indicates  that 
generative  activities  in  general  may  benefit  learning  (reading  comprehension)  but  may  inhibit  navigational  hypertext 
when  learners  are  engaged  in  problem  solving.  Generative  activities  may  produce  interference  with  respect  to  their 
navigational  choice  and  sequencing  of  text.  When  navigating  through  the  hypertext  to  solve  a problem  has  already 
posed  considerable  difficulties,  the  addition  of  generative  activities  proves  to  be  a distraction  and  an  overwhelming 
burden  in  problem  solving. 

In  contrast  to  the  results  of  Experiment  One  on  problem-solving  scores,  we  found  that  students  assigned  the 
Generative  condition  outperformed  those  assigned  the  Non-generative  condition.  We  had  expected  that  individuals 
assigned  the  Linear  condition  would  do  better  than  those  assigned  the  Navigational  condition.  While  it  was  true,  the 
analysis  failed  to  reach  a statistical  significance  level.  A close  look  at  the  raw  reading  comprehension  score  (reading 
comprehension  score  before  being  converted  into  efficient  score)  revealed  that  learners  assigned  linear  condition 
outperformed  those  assigned  navigational  condition  (F  = 4.745,  P <.05).  This  finding  indicates  that  when  students' 
raw  reading  comprehension  scores  were  converted  into  efficient  score  (raw  score/dwell  time),  their  dwell  time  wiped 
out  most  of  the  effect  (see  table  1 below).  They  had  high  reading  comprehension  score  but  were  not  efficient  because 
they  spend  more  time  and  therefore  they  reviewed  significant  number  of  facts. 

Table  1.  Means  and  Standard  Deviations  of  Dwell  Time  and  Raw  Reading  Comprehension  Score  by  Students 
in  the  Two  Conditions  Using  Hypertext  for  Reading  Comprehension  Test 


Dwell  Time  (minutes) 

Raw  Score 

CONDITIONS 

M 

SD 

M 

SD 

Linear  Condition 

36.89 

12.52 

20.02 

4.91 

Navigational  Condition 

28.18 

11.87 

16.44 

5.04 

Conclusion  and  Implications 

In  this  study  we  examined  students*  problem  solving  and  reading  comprehension  when  using  text  in  linear, 
navigational,  linear  generative  or  navigational  generative  hypertext  system  to  solve  a problem  or  to  prepare  for  a 
reading  comprehension  test.  The  results  support  the  driving  force  of  learners'  goals,  whether  implicit  in  the 
educational  setting  (e.g.,  studying  for  a test)  or  adopted  from  a contrived  context  (e.g.,  anchored  instruction  or 
problem-based  learning),  in  co-determining  the  learning  context.  Understanding  students'  goals  in  turn  explains  the 
outcomes  of  instruction,  the  effects  of  instructional  designs  (e.g.,  learner  control/program  control).  Further  the  study 
also  provides  empirical  evidence  regarding  which  formats  may  best  be  used  to  teach  students  problem  solving  and/or 
comprehension  skills.  For  example,  students  with  problem  solving  goals  who  used  the  Navigational  hypertext  system 
outperformed  those  students  assigned  the  Linear  conditions.  However,  when  students  had  reading  comprehension 
goals,  students  using  Generative  hypertext  system  outperformed  those  using  Non-generative  hypertext  system.  We 
did  not  find  significant  difference  between  linear  and  navigational  activities  with  respect  to  reading  comprehension, 
being  a finding  that  could  be  attributed  to  the  effect  of  dwell  time.  Although  we  are  still  not  sure  why  generative 
activities  did  not  benefit  learning  overall  we  did  find  that  generative  activities  hurt  learners  when  they  were  given 
more  freedom  to  navigate  to  solve  a posed  problem. 

Given  the  size  and  complexity  of  many  hypertext  environments  that  students  are  expected  to  use  (e.g.,  the 
World  Wide  Web,  CD-ROMs),  it  is  imperative  that  educators  consider  the  goals  that  students  have  for  engaging  in 
these  environments.  Educators  exercise  some  control  over  the  goals  that  students  adopt  by  creating  a context  in 
which  students  read  for  a test  or  read  to  complete  some  performance-based  assignment  such  as  problem  solving.  In 
so  doing,  they  must  recognize  that  materials  optimized  for  some  goals  (e.g.,  information  search  and  retrieval)  may  be 
distracting  and  clumsy  for  others  (e.g.,  reading  comprehension).  This  is  consistent  with  the  theories  of  situated 
cognition  and  ecological  psychology  in  which  it  is  posited  that  different  learner  goals  establish  different  learning 
contexts,  which  may  or  may  not  couple  the  learner  and  environment.  While  the  literature  on  both  generative  activity 
and  learner  control  does  not  lead  to  a clear  conclusion  regarding  the  usefulness  of  learner  control  as  an  instructional 
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strategy  and  generative  activity  as  a format  of  hypertext,  the  explanation  may  lie  in  the  goals  that  users  adopt  for 
learning.  This  study  provided  some  empirical  evidence  that  learner’s  goals  must  be  understood  to  fully  explain  the 
interaction  of  student  with  instruction.  This  study  found  that  learners  in  the  Generative  condition  who  had  reading 
comprehension  goal  outperformed  those  assigned  the  Non-generative  condition.  Learners  with  problem-solving  goals 
scored  higher  on  a problem-solving  measure  than  did  learners  with  less  control.  Such  a difference  was  true  but  was 
not  significant  when  learners  had  reading  comprehension  goals.  This  finding  is  of  interest;  especially  when  their 
reading  efficiency  score  is  taken  into  consideration.  Future  research  needs  to  continue  to  explore  the  interaction  of 
goals  and  instructional  context,  especially  with  respect  to  the  instructional  activities  that  require  more  mindful 
engagement,  such  as  reading  comprehension  test  focus  on  inferential  information  rather  than  factual  information. 
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Abstract 

The  purpose  of  this  study  was  to  investigate  the  process  used  by  learners  to  direct , locate , and  integrate 
information  resources  for  use  in  a project-based  environment.  Four  cases  (N=9)  were  analyzed  from  an 
introductory  educational  technology  course  during  a unit  on  telecommunications.  Participants  were  asked 
to  generate  projects  for  integrating  the  Internet  into  the  curriculum.  Within  this  project-based  context , 
learners  searched  for  information  resources  that  would  accompany  their  project  ideas.  Three  major 
findings  related  to  use  of  hypermedia  systems  during  project-based  learning  were  discussed:  (a) 

progressing  from  data-driven  to  goal-driven  approaches  were  critical  to  developing  a coherent  project 
idea ; (b)  consolidating  multiple  information  resources  with  proposed  project  methods  and  rationales  was 
challenging  for  learners;  and  (c)  metacognitive , domain , and  system  knowledge  appear  critical  to 
achieving  coherence  in  project  development.  Implications  related  to  the  role  of  instructional  scaffolding  in 
encouraging  goal-driven  and  metacognitive  processing  during  open-ended  learning  are  considered. 


Introduction 

In  the  information  age,  it  has  become  increasingly  important  for  students  to  learn  how  to  identify 
information  needs,  locate  corresponding  information  resources,  and  integrate  information  from  a variety  of  sources 
into  cogent,  productive  uses  (Moore,  1995).  Such  requirements  represent  radically  different  cognitive  goals  than  are 
typically  fostered  in  traditional  classrooms  - goals  exemplified  by  teacher-centered  or  instruction-centered 
interactions  that  prescribe  responses  from  learners.  Externally-centered  instructional  methods,  according  to  critics, 
fail  to  support  meaningful  generation  of  understanding  (Jonassen,  1991)  and  fail  to  address  the  knowledge 
requirements  of  a rapidly  expanding  technological  society  (Hannafin  & Land,  1997). 

Yet,  recently,  much  attention  has  been  dedicated  to  diverse  applications  of  technology  that  support  active, 
student-centered  learning.  Contemporary  theoretical  perspectives  have  emerged  that  emphasize  the  centrality  of  the 
learner  in  exploring,  manipulating,  and  generating  understanding  (Cognition  and  Technology  Group  at  Vanderbilt, 
1990;  Spiro,  Feltovich,  Jacobson,  & Coulson,  1991).  Open-ended  learning  environments  (OELEs)  are  learner- 
centered  environments  that  facilitate  the  unique  efforts  of  learners  to  generate  and  refine  meaning  (Hannafin,  Hall, 
Land,  & Hill,  1994).  They  represent  a variety  of  approaches  and  technologies  that  enable  learners  to  engage 
cognitively  complex  tasks  that  require  critical  thinking,  self-direction,  problem-solving,  or  meaningful  integration  of 
knowledge  (Land  & Hannafin,  1996). 

Project-based  learning  is  an  approach  to  open-ended  learning  that  encourages  meaningful  learning  through 
student-directed  investigation  (Blumenfeld,  Solo  way,  Marx,  Krajcik,  Guzdial,  & Palincsar,  1991).  Students  pursue 
solutions  to  open-ended  problems  by  formulating  questions  for  investigation,  designing  plans  or  proposals,  collecting 
and  analyzing  information,  and  creating  “artifacts”  or  products  of  their  understanding  (Blumenfeld  et  al.,  1991). 
Project-based  approaches  satisfy  two  primary  components:  (a)  learners  generate  a question  or  problem  that  serves  to 
organize  and  drive  learning  needs;  and  (b)  learners  produce  a final  product  or  series  of  products  to  address  the 
driving  question/problem  derived  (Blumenfeld  et  ah,  1991).  Through  extended  investigation  and  experience  with  the 
problems  under  study,  learners  are  presumed  to  identify  knowledge  needs,  locate  required  resources  to  help  them 
meet  these  needs,  and  integrate  diverse  information  into  coherent  explanations  or  projects. 

With  project-based  approaches,  information  resources,  such  as  the  World  Wide  Web,  can  play  a critical 
role  in  the  learning  environment.  In  the  process  of  working  on  projects,  students  require  access  to  information.  As 
students  generate  learning  goals  or  problems,  they  seek  new  information  when  they  find  themselves  unable  to 
proceed  without  deepening  understanding  of  what  is  already  known  (cf.,  Moore,  1995).  Identifying  and  generating  a 
“need  to  know”  is  assumed  critical  for  meaningful  integration  of  information.  Organizing  frameworks,  such  as 
student-generated  problems  or  questions,  are  required  for  learners  to  focus,  verify,  and  generate  cohesion  to  the 
information  retrieved  (Hill  & Hannafin,  1997;  Moore,  1995). 

Simply  providing  access  to  information  does  not  guarantee  if  will  be  useful  to  students  in  generating  and 
refining  projects.  That  is,  students  might  search  vast  databases  of  information,  but  fail  to  use  this  information  to 
solve  problems  or  deepen  understanding.  The  process  would  be  similar  to  searching  for  information  with  no  purpose, 
intent,  or  reason.  Without  generating  driving  questions  or  learning  goals,  it  is  unlikely  that  retrieved  information 
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would  be  meaningful  to  learners,  since  it  would  be  isolated  from  a meaningful  context  (cf.,  Brown,  Collins,  & 
Duguid,  1988). 

Yet,  the  process  whereby  learners  generate  information  needs,  search  information  resources,  and  integrate 
varied  resources  into  a cohesive  project  is  little  understood.  What  is  known,  suggests  that.leamers’  level  of 
processing  is  often  poorly  matched  to  the  processing  needs  of  information-rich  OELEs.  For  example,  previous 
research  on  student  learning  with  OELEs  indicates  that  learners  often  have  difficulty  generating  unique  learning 
goals  and,  instead,  rely  heavily  upon  external  guidance  to  direct  their  activity  (Land  & Hannafin,  1997;  Oliver, 

1999).  Furthermore,  studies  of  learner  use  of  the  World  Wide  Web  have  indicated  that  students  frequently  fail  to 
establish  or  alter  task-relevant  goals,  and  they  show  little  evidence  of  engaging  meaningful,  reflective  activity  (Hill 
and  Hannafin,  1997).  Individual  knowledge  variables,  such  as  metacognition,  system  knowledge,  and  prior  domain 
knowledge  have  been  shown  to  influence  the  extent  to  which  learners  productively  seek,  locate,  and  integrate 
information  resources  (Hill  & Hannafin,  1997).  Qualitative  studies  of  children’s  use  of  information  retrieval  systems 
during  project-based  learning  show  that  students  often  (a)  fail  to  generate  questions  that  extend  from  what  is  known; 
(b)  are  ineffective  in  using  search  terms;  and  (c)  process  accessed  information  as  “answers,”  rather  than  as  data  for 
reflection  and  evaluation  (Moore,  1995;  Wallace  & Kupperman,  1997). 

Accordingly,  the  purpose  of  this  research  was  to  investigate  the  process  used  by  learners  to  guide,  locate, 
and  integrate  information  resources  for  use  in  a project-based  environment.  Addressing  this  purpose  requires  insight 
into  the  unique  intentions,  actions,  and  sense-making  processes  of  learners  though  out  the  learning  process.  The 
goals  of  this  research  were  primarily  descriptive  in  nature  and  were  aimed  at  gaining  a deeper  understanding  of  the 
problems  and  processes  experienced  by  learners  in  this  endeavor. 

The  study  examined  the  following„questions: 

1 . What  general  strategies  do  participants  use  to  guide  their  information  seeking? 

2.  What  is  the  role  of  system,  domain,  and  metacognitive  knowledge  in  locating  web  resources? 

To  what  extent  do  students  integrate  web  resources  into  a coherent  project? 


Method 


Nine  undergraduate  pre-service  teachers  participated  in  the  study.  The  participants  were  primarily  attaining 
certification  in  early  childhood  and  elementary  education,  although  three  were  pursuing  secondary  English 
certification.  They  were  enrolled  in  a required  course,  “Learning  with  Educational  Technologies,”  and  were  awarded 
extra  credit  for  participating  in  the  study.  Students  were  asked  to  generate  project  ideas  for  an  instructional  planning 
activity  to  integrate  the  Internet  into  the  classroom.  Within  this  project-based  context,  learners  searched  for 
information  resources  that  would  accompany  their  project  ideas.  The  participants  elected  whether  to  work  as  groups 
or  by  themselves  for  the  project.  The  groups  and/or  individuals  were  studied  as  separate  cases  and  analyzed  for 
similarities  and  differences  by  referencing  an  existing  theory-based  pattern  (Yin,  1994). 

Three  primary  techniques  were  used  for  data  collection:  Think-aloud  protocols,  videotaped  observations  of 
system  use,  and  student-generated  documentation  (i.e.,  final  project  documentation  and  history  lists  of  web  usage). 
Data  collection  lasted  four  consecutive  days.  Each  class  period  lasted  approximately  2.5  hours.  All  interactions  were 
videotaped,  with  the  camera  recording  activities  on  the  screen  and  audio  recording  student  conversations.  The 
instructor  and  another  co-investigator  were  present  during  all  class  periods  and  continually  circulated  among  the 
students  to  facilitate  their  learning. 

Think-aloud  protocols  and  observations  of  videotaped  system  events  yielded  foundation  data  for  the 
analysis.  Think-aloud  protocols  were  transcribed  and  matched  to  videotaped  observations  of  learner  actions  with  the 
web.  Data  were  collated  according  to  each  of  three  pre-defined  research  questions:  Strategy  use,  information 
location,  and  coherence  of  project  ideas  (see  for  example,  Merriam,  1988).  Criteria  were  developed  for  each 
question,  which  were  used  to  assign  data  (see  Lincoln  & Guba,  1985).  For  instance,  isolated  responses  such  as 
"maybe  I should  try  another  search  engine,"  or  "I  need  to  find  something  on  public  schools  on  the  web"  were  labeled 
as  individual  units  (Ericsson  & Simon,  1992).  Both  authors  coded  the  data  in  order  to  achieve  100%  agreement  on 
the  codes. 

Findings  and  Implications 

Question  1:  General  Strategies  Associated  with  Information  Seeking 

This  question  examined  the  strategies  used  by  participants  to  identify  knowledge  deficiencies,  set  learning 
goals,  and  seek  information  using  the  World-Wide  Web  to  generate  and  refine  project  ideas.  Each  protocol  was 
coded  for  evidence  of  two  categories  of  strategies: 

1.  data-driven  strategies; 

2.  goal-driven  strategies  (adapted  from  Hmelo,  in  press). 
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Category  1:  Data-driven  strategies.  Data-driven  strategies  involved  inferring  from  the  data  (i.e.,  web 
resources)  to  possible  project  ideas  or  explanations  for  use  of  that  data.  Using  this  approach,  participants  identified 
broad  subject  areas,  conducted  a search,  read  information  on  the  topic,  and  formulated  ideas  and  questions  from  the 
web  resources.  During  initial  stages  of  project  planning,  all  participants  used  data-driven  strategies  to  explore 
possibilities  for  ideas.  These  strategies  were  most  readily  apparent  in  instances  where  participants  conducted  a 
search  on  a general  topic  (e.g.,  the  state  of  California)  and  accessed  a number  of  possible  links  from  a given  page 
(e.g.,  tourism  in  California)  or  list  of  search  results.  They  would  then  “browse”  the  sites  that  appeared  interesting  in 
order  to  evaluate  their  usefulness.  Janie,  for  instance,  conducted  a search  to  find  sites  affiliated  with  grocery  stores 
for  her  project  idea.  She  stated,  “Let’s  see  if  something  about  grocery  stores  is  listed.. .[finds  Groceries  On-line  link]. 
Let’s  find  out  about  this  website.”  She  read  the  information,  returned  to  the  retrieved  search  list,  and  began 
exploring  other  grocery  store  links  such  as  Food  Lion  and  Shaw  Supermarkets.  The  function  of  these  searches  was 
to  focus  ideas  and  to  “see  what  is  out  there”  before  committing  to  a specific  project  idea. 

Data-driven  approaches  were  marked  by  interpretations  of  the  significance  of  isolated  data  (e.g.,  a specific 
site  accessed)  independent  of  pre -defined  and  specific  contexts  for  using  them.  Such  approaches  did  not  require 
deliberate  search  strategies  as  a means  for  finding  information.  Rather,  participants  identified  broad  topics,  used 
general  search  terms  (e.g.,  “travel”,  “groceries”,  “Shakespeare”)  viewed  “hit  lists”  of  retrieved  sites,  and  browsed 
through  the  sites  in  an  effort  to  find  ones  that  could  be  used  for  the  project  idea.  Janie,  for  instance,  would  often 
browse  through  long  lists  of  links,  without  specifying  what  she  was  trying  to  accomplish:  “OK,  I’m  getting  to  the  end 
of  the  list  of  366  sites.  And  it’s  always  the  last  page  ...  that  has  the  information  you’re  looking  for...” 

Participants  who  relied  heavily  on  this  approach  characteristically  sorted  through  many  irrelevant  sources 
that  ultimately  had  little  to  no  bearing  on  developing  project  ideas;  instead,  they  engaged  in  tangential  inquiry  of  web 
resources  without  a clear  “anchor”  or  context  for  directing  the  search  task.  While  data-driven  reasoning  was  common 
among  all  cases  during  initial  stages  of  project  formulation,  two  cases  relied  heavily  upon  data-driven  strategies 
throughout  the  entire  project  (Case  3:  Janie,  and  Case  4:  [Desi,  Nell,  and  Opal]).  Their  reliance  on  external  data  as 
the  driving  force  for  ideas  often  led  to  problems  in  locating  information  and  in  discerning  how  information  could  be 
useful.  In  turn,  this  often  led  to  frustration,  inefficiency  in  information  retrieval,  and  the  use  of  frequent  statements 
such  as  “We’re  just  not  finding  anything.” 

Yet,  use  of  data-driven  strategies  sometimes  resulted  in  serendipitous  findings  that  led  the  participants  in 
new  directions  for  their  project  ideas.  For  instance,  when  browsing  the  EduStock  site,  Kara  and  Tammy  found  a 
stock  market  simulator  that  enabled  real-time,  simulated  buying  and  selling  of  stocks.  They  incorporated  this  feature 
into  their  project  idea,  which  prior  to  finding  this  site,  was  more  vague  and  disconnected  (“We’re  going  to  do 
something  with  the  stock  market”).  Similarly,  Tammy  noted  that  the  stock  market  is  in  the  process  of  converting 
from  decimals  to  fractions.  She  noted,  “We  can  integrate  more  math  skills,  since  the  kids  will  need  to  convert.” 

Their  project  ideas  were  formulated  and  expanded  based  upon  the  identification  of  resources  and  tools  that  they  did 
not  intentionally  access  for  use  in  the  project. 

Similarly,  when  Janie  was  browsing  grocery  store  sites  for  cost  comparison  data,  she  found  a link  to  a 
supermarket  chain  in  Hong  Kong.  She  said,  "a  Web  site  from  Hong  Kong  I can  go  grocery  shopping  on.  So,  we  get 
to  expand  our  project  to  include  different  countries. . . it  has  its  prices  in  their  currency.  Boy  my  kids  are  going  to 
have  to  do  some  big  time  conversions."  In  both  of  these  examples,  the  participants  had  not  previously  considered 
these  ideas,  and  they  were  able  to  identify  resources  and  consider  alternatives  for  using  them.  Use  of  data-driven 
strategies  led  to  expansion  of  possibilities  for  the  web  project,  in  that  the  search  space  was  not  constrained  by 
requirements  for  specific  forms,  or  purposes,  of  data. 

Category  2:  Goal-driven  strategies.  With  goal-driven  approaches,  participants  sought  information  in  the 
context  of  a goal,  hypothesis,  or  question.  Such  strategies  required  a judgment  regarding  what  information  was 
desired,  prior  to  knowing  what  information  was  available.  When  searching,  participants  considered  the  kinds  of  web 
resources  that  were  needed,  bearing  in  mind  both  the  project  focus  and  resource  needs,  rather  than  collecting  isolated 
sets  of  web  resources.  Goals  for  information-seeking  were  inextricably  tied  to  learner  hypotheses  and/or  “needs  to 
know”  for  the  project  idea.  A major  function  was  validating  or  testing  the  viability  of  project  ideas. 

Goal-driven  approaches  represented  strategic,  directed  intentions  to  access  specific  sites  or  information. 

The  purpose  of  these  goals  was  to  confirm  that  specific  information  was  accessible  in  order  to  apply  it  to  the  project 
idea.  These  strategies  were  most  obvious  in  cases  where  the  leamer(s)  were  looking  for  a specific  site  or  piece  of 
information.  To  illustrate,  Case  1 participants  proposed  a project  for  elementary  students  to  learn  geography  by 
planning  a virtual  vacation.  They  identified  the  State  of  California  as  a possible  place  to  visit.  Accordingly,  Jill 
suggested,  “Let’s  see  if  we  can  get  a Disneyland  [site]  to  come  up.  Because  if  we  can’t  find  it,  the  kids  won’t  be  able 
to  access  it.”  They  continued  these  types  of  directed  searches  for  airline  and  rental  car  prices,  gasoline  prices,  maps 
of  the  areas,  and  so  on.  The  purpose  of  these  searches  was  to  determine  whether  or  not  the  idea  was  viable  using 
web  resources.  If  they  were  unable  to  access  the  information  and/or  find  it  useful,  it  was  unlikely  that  their  students 
would  benefit  from  the  activity.  Finding  and  validating  the  usefulness  of  specific  sites  or  ideas  assisted  participants 
in  expanding  or  filling  in  the  details  of  their  project  ideas. 
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Use  of  goal-driven  strategies  required  a focus  that  helped  participants  to  limit  the  search  space  and  to  filter 
relevant  from  irrelevant  information.  When  participants  approached  the  search  task  from  the  perspective  of 
confirming  a hypothesis,  they  were  able  to  apply  the  retrieved  information  to  solve  a problem  that  was  relevant  to 
their  project  idea.  For  instance,  Case  1 (Jack,  Jane,  & Jill)  were  looking  for  sites  that  their  students  could  use  to  plan 
and  budget  a virtual  family  vacation.  Jane  commented,  “We  also  have  to  figure  out  gas  prices.  It’s  got  to  be  more 
expensive  at  different  [geographic]  locations.”  Formulating  this  hypothesis  led  the  group  to  investigate  gasoline 
prices  in  different  states  across  the  United  States.  The  information  retrieved  was  then  useful  for  helping  them  to 
evaluate  their  project  focus  and  consider  the  problem  stipulations  that  would  be  relevant  to  their  future  students. 

Similarly,  Case  2 (Tammy  and  Kara)  considered  a project  idea  for  their  students  to  use  on-line,  simulated 
investment  portfolios  as  a means  to  pay  for  college.  When  looking  for  information  on  tuition  costs  at  the  University 
of  Oklahoma  website,  Tammy  noted:  ‘This  tells  us  all  of  the  fees  per/credit  hour.  So,  an  Ivy  League  school  would 
cost  a lot  more.  We  could  [look  up  cost  information  for  different  schools]  and  give  them  a choice  between  attending 
a community  college,  OU,  or  an  [Ivy  League]  School.”  Following  up  on  this  hypothesis  about  differential  cost 
structures  enabled  them  to  find  information  that  established  another  dimension  of  complexity  to  their  project.  When 
search  strategies  were  driven  by  hypotheses,  the  accessed  information  did  not  remain  isolated  and  discrete;  rather,  it 
was  situated  within  an  applied  context  (Brown  et  al.,  1988);  in  this  case,  the  project  context  that  was  generated  by  the 
participants. 

Question  2:  Knowledge  Sources  for  Retrieving  Information  from  the  WWW 

Question  one  examined  the  general  strategies  used  by  participants  to  define  the  information  they  wanted  to 
retrieve  from  the  Web  for  use  in  the  project.  Question  two  extends  from  this  question  to  examine  how  successful 
participants  were  in  actually  retrieving  or  locating  this  information.  Three  sources  of  knowledge  influence  successful 
retrieval  of  information:  (1)  system  knowledge  (and  use  of  search  terms);  (2)  the  role  of  domain  knowledge  in 
locating  information;  and  (3)  metacognitive  knowledge  of  how  to  monitor  and  revise  search  techniques  (Hill  & 
Hannafin,  1997).  It  is  important  to  note  that  all  three  categories  of  system,  metacognitive,  and  domain  knowledge 
operate  simultaneously  to  influence  successful  retrieval  of  information.  Yet,  for  the  purposes  of  distinguishing  their 
respective  influences,  they  will  be  treated  separately  in  the  reporting  of  results. 

System  knowledge.  Participants  typically  retrieved  information  from  the  web  by  using  search  engines, 
search  engine  indexes,  and/or  known  URLs  that  were  typed  into  the  “go  to”  field.  When  participants  were  already 
aware  of  sites  or  features  available  on  the  Web,  they  were  successful  in  locating  them  and  assigning  a use  for  them  in 
the  project.  For  instance,  Jack  from  Case  1 [trip-planning  topic],  was  already  aware  of  many  sites  on  the  Web  that 
could  be  useful  for  their  project.  He  noted,  “There  is  a deal  on  Yahoo  that  can  help  you  make  maps.  We  can  have  the 
students  use  this  to  make  maps  of  where  they  will  go.  [The  map  tool  on  Yahoo]  will  tell  them  how  long  it  will  take 
to  [travel  to  their  destination]  and  will  help  keep  them  realistic.”  In  this  instance,  awareness  of  features  available  on 
the  web  and  a purpose  for  how  they  could  be  used  helped  Case  1 successfully  locate  needed  resources  for  their 
project. 

Whereas  system  knowledge  of  how  to  use  these  search  features  was  critical  for  locating  information,  it  was 
also  clear  that  the  way  in  which  questions  were  asked  influenced  the  search  terms  defined  which  influenced 
subsequent  retrieval.  For  example,  Janie,  who  reported  to  be  an  experienced  Web  user,  continued  to  use  vague, 
broad  search  terms  like  ’’schools"  and  "web".  When  she  encountered  hundreds  of  retrieved  sites,  she  proceeded  to 
systematically  browse  through  all  the  abstracts  related  to  those  sites,  stating  that  the  best  one  will  surely  be  found  on 
the  last  page  of  abstracts.  Standard  Boolean  operators  were  not  typically  used  by  Janie  to  refine  or  narrow  the  scope 
of  a search.  Consequently,  she  frequently  reported  being  frustrated  at  failing  to  find  useful  sites.  In  one  critical 
instance,  Janie  neglected  to  use  the  bookmark  feature  to  store  sites  of  interest  to  her.  When  she  wanted  to  return  to 
these  sites  on  the  second  day,  she  spent  almost  2 hours  searching  for  them  (unsuccessfully). 

Case  4 (Desi,  Nell,  and  Opal)  experienced  similar  difficulties  with  identifying  and  using  focused  search 
terms.  In  one  interaction,  Desi  remarked:  “We’re  trying  to  find  information  on  history.  I’m  trying  to  think  of  how  we 
could. ..word  it.  Because  we’re  not  wanting  what  happened  to  Shakespeare,  we’re  wanting  what  happened  in  general 
history,  right?  Just  try  searching  for  “16th  century.” 

The  vague  nature  of  the  search  terms  used  by  Case  4 was  common  throughout  their  web  searching. 
Consequently,  they  found  little  information  of  use  to  them  and  frequently  remarked  they  nothing  was  available  on  the 
web  on  “history.”  Throughout  the  entire  period  in  which  they  worked  on  the  project,  Case  4 failed  to  delineate  the 
aspects  of  Shakespeare’s  life  or  works  that  would  be  the  focus  of  the  project.  Nor  did  they  define  a specific  historical 
event  or  artistic  influence  that  was  considered  relevant  to  his  writing.  Consequently,  they  used  equally  ambiguous 
search  terms  such  as  “17lh  century”  or  “Shakespeare  history”  in  an  effort  to  retrieve  poorly  specified  information. 

Domain  knowledge  of  the  topic  under  investigation.  Learner  domain  knowledge  has  been  found  to  be  a 
critical  factor  for  both  developing  instructional  contexts  and  for  locating  information  in  hypermedia  information 
systems  (Hill  & Hannafin,  1997).  Making  meaning  from  new  knowledge  is  largely  influenced  by  the  extent  to 
which  it  is  connected  to  existing  knowledge  (Brown,  Bransford,  Ferrara,  & Campione,  1983).  In  this  study,  we 
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found  that  limitations  in  domain  knowledge  were  difficult. to  overcome  in  an  ill -structured  information  system  where 
inquiry-based  searches  were  required.  To*  illustrate,  many  of  the  problems  experienced  by  Case  4 in  finding 
information  seemed  to  be  driven  by  a lackof  domain  knowledge  about  the  time  period  and  historical  events 
associated  with  Shakespeare’s  writing.  In  some  instances,  their  questions  were  based  on  misunderstandings  or 
inaccurate  subject  knowledge,  and  they  were  often  unaware  of  such  misunderstandings.  This  led  to  a snowball  effect 
of  frustrations,  since  they  then  had  difficulty  finding  information  and  failed  to  consider  that  their  initial  assumptions 
were  inaccurate. 

In  other  instances,  however.  Case  4 occasionally  found  (and  learned  about)  information  that  triggered 
awareness  of  their  misunderstandings.  For  instance,  after  two  days  of  unsuccessful  browsing  about  history  and 
Shakespeare,  they  stumbled  upon  a site  that  provided  information  regarding  the  reign  of  Elizabeth  I.  Nell  noted,  “If 
we  look  up  Elizabeth  I,  that  would  tell  us  the  history  there,  and  what  was  going  on  when  he  was  [in  England]  because 
it  says  that  his  writing  got  gloomier  during  that  time.” 

Similarly,  Case  4 had  misunderstandings  regarding  the  time  period  in  which  Shakespeare  was  writing,  and 
consequently  used  inaccurate  search  terms.  During  day  2,  the  group  searched  for  historical  information  on  the 
Middle  Ages.  Later,  they  saw  a title  on  a search  index  entitled  ‘The  Renaissance  and  the  Globe  Theater”.  Nell 
noted,  “Renaissance!  That  is  why  all  of  this  Middle  Ages  stuff  was  not  coming  up.  So  let’s  look  up  Renaissance 
now.”  These  instances  illustrate  how  lack  of  domain  knowledge  seriously  hindered  the  extent  to  which  participants 
could  productively  find  relevant  information  on  the  Web.  However,  it  also  appears  that  participants  can  use  an 
information  system  such  as  the  Web  to  recognize  limitations  in  knowledge  and  to  use  what  is  learned  as  the  basis  for 
further  investigation. 

In  contrast.  Case  1 experienced  few  problems  with  locating  relevant  information.  We  believe  that  this  was 
due,  in  part,  to  the  fact  that  all  members  of  this  group  had  well-established  schemata  for  trip  planning  and  were  thus 
able  to  draw  upon  this  knowledge  to  ask  questions  and  use  search  terms  effectively.  For  instance.  Jack  commented, 
“We  haven’t  looked  at  [websites]  for  airline  tickets  and  car  rentals.  Go  to  Hertz.com.”  Later,  they  used  the  Yahoo 
Travelocity  site  for  students  to  investigate  prices  for  airline  reservations.  In  this  instance,  prior  knowledge  and 
experience  with  trip  planning  (as  well  as  system  knowledge  of  what  was  available  at  Yahoo  and  the  Hertz  car  rental 
home  page)  made  locating  relevant  information  productive  for  this  group.  Prior  domain  knowledge,  then,  was 
critical  for  identifying  a “need  to  know”  and  for  then  using  search  terms  and  indexes  to  retrieve  the  needed 
information. 

Metacognitive  Knowledge.  Metacognitive  knowledge  is  often  characterized  as  consisting  of  three 
interrelated  components:  (a)  awareness  of  one’s  own  cognition  and  the  degree  to  which  one’s  cognitive  efforts  are 
successful;  (b)  knowledge  about  the  different  cognitive  demands  of  different  learning  tasks,  and  (c)  procedural 
knowledge  of  strategies  to  employ  when  the  status  quo  is  not  successful  (Gamer  and  Alexander,  1987;  Moore, 

1995).  In  the  context  of  our  study,  metacognitive  knowledge  refers  to  the  process  of  reflecting  on,  or  monitoring  the 
effectiveness  of  the  search  process  and  then  refining  the  process  when  necessary.  Previous  research  on  use  of  the 
World  Wide  Web  found  that  metacognitive  knowledge  was  critical  for  participants  “to  reflect  on  their  search 
processes,  refine  their  actions,  and  make  better  use  of  the  system”  (Hill  & Hannafin,  1997,  p.  56).  Participants  with 
low  metacognitive  knowledge  often  used  unproductive  search  strategies  and  failed  to  monitor  and  alter  those 
strategies  when  necessary  (Hill  & Hannafin,  1997). 

All  four  of  our  cases  demonstrated  some  reflection  and  monitoring.  At  some  point,  the  participants  in  each 
case  reflected  on  the  effectiveness  of  their  search  strategies;  in  three  of  the  cases,  the  participants  acknowledged  that 
their  search  efforts  were  not  fruitful.  However,  as  we  discuss  below,  our  data  demonstrate  that  awareness  alone  does 
not  constitute  a sufficient  condition  for  metacognitive  knowledge.  The  role  of  learner  metacognitive  knowledge  in 
locating  desired  resources  was  apparent  in  instances  where  participants  refined  search  terms  and  strategies  when 
needed.  Case  1,  for  instance,  often  evaluated  the  limitations  of  the  accessed  information  and  re-focused  their  search 
efforts  to  narrow  the  type  of  information  retrieved.  Using  the  WWW  to  effectively  locate  information  requires  key 
components  of  metacognition  - continual  monitoring  of  learning  activities  and  goals,  and  an  awareness  of  instances 
when  plans  are  inconsistent  with  outcomes. 

Case  2 (Tammy  and  Kara)  also  relied  on  metacognition  to  re-direct  the  search  process.  While  searching  for 
performance  data  on  the  company  Kimberly  Clark  from  the  EduStock  site,  they  found  a website  that  they  initially 
hoped  would  provide  this  information.  Instead,  the  website  was  irrelevant  for  their  project.  Tammy  noted,  ‘This 
website  gives  me  a ton  of  stuff.  But,  I want  [a  web  page  about]  the  company  Kimberly  Clark.  I’m  sure  they  have 
their  own  web  page.”  She  then  revised  her  search,  using  a Yahoo  index  and  found  the  home  page  for  the  company. 

Effective  location  of  information  from  the  WWW  seems  to  require  the  following:  (a)  awareness  of  the  form 
and  purpose  of  information  desired;  (b)  awareness  of  whether  the  accessed  information  matches  the  desired 
information  (i.e.,  filter  relevant  from  irrelevant  information);  and  (c)  awareness  of  how  to  refine  search  terms  and 
strategies  in  order  to  produce  effective  results.  Not  all  participants  used  these  processes,  however,  and  those  who  did 
not  experienced  difficulties  in  accessing  desired  information.  For  instance.  Case  4 was  searching  for  poorly 
specified  information  for  their  Shakespeare  project  (i.e.,  world  events  associated  with  the  time  of  his  writing).  When 
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they  recognized  that  the  information  accessed  did  not  match  what  they  wanted,  they  failed  to  refine  either  their 
search  goals  or  their  search  strategies.  Their  main  problem  seemed  to  be  that  they  never  identified  what  information 
they  actually  wanted.  Without  clear  criteria  for  relevancy,  they  were  unable  to  effectively  sort  through  what  they 
found.  They  repeatedly  entered  the  same  search  terms,  often  switching  search  engines,  with  equally  unproductive 
results.  In  most  instances,  changing  search  engines  appeared  to  be  an  ineffective  strategy  for  refining  a search.  The 
failure  of  some  participants  to  recognize  the  ineffective  outcomes  of  this  strategy  seemed  often  to  be  related  to 
inadequate  system  knowledge.  They  did  not  know  what  else  they  could  do  when  their  known  and  preferred  strategies 
were  limited. 

For  Case  4,  the  failure  to  demonstrate  metacognitive  knowledge  concerning  search  results  seemed  explained 
by  inadequate  domain  knowledge,  in  addition  to  insufficient  system  knowledge.  In  fact,  based  on  three  of  our  cases, 
we  might  infer  that  effective  metacognition  depends  on  adequate  knowledge  in  the  other  two  areas.  For  example,  the 
two  cases  that  seemed  to  demonstrate  low  metacognitive  knowledge.  Case  3 (Janie)  and  4 (Opal,  Nell  and  Desi), 
were  also  cases  in  which  other  knowledge  was  low  and  the  goal  for  searching  remained  ill  defined.  Case  1 
demonstrated  high  metacognitive  knowledge  and  also  demonstrated  high  system  and  domain  knowledge.  By 
contrast,  however.  Case  2 (Tammy  and  Kara)  also  demonstrated  high  metacognitive  knowledge  but  had  only  novice 
level  system  knowledge  and  low  domain  knowledge  (their  domain  was  the  stock  market).  What  seemed  to  separate 
Case  2 from  both  Case  3 and  4 was  their  continual  monitoring  of  the  search  process  in  light  of  their  evolving  goal. 
The  following  exchange  captures  how  they  engaged  in  this  process. 

Tammy:  This  is  forecasting  college  costs.  They  can  come  to  this  site. 

Kara:  So  are  we  still  doing  the  stock  thing?  Incorporating  both  or  changing? 

Tammy:  This  is  why  they  are  doing  the  stock  market.  You  need  to  go  to  college  and  figure  out  how  much  it  costs 
you,  you  need  to  go  out  and  figure  how  to  invest  to  make  this  much  money  so  you  can  plan  for  college 
Kara:  That's  a good  idea;  point  for  making  money.  That  makes  sense. 

For  Case  2,  metacognition  was  seemingly  able  to  compensate  for  low  domain  and  moderate  system 
knowledge.  For  Cases  3 and  4,  the  benefits  of  their  metacognitive  processing  seemed  to  be  limited  by  the  absence  of 
well-defined  project  ideas  or  even  search  goals.  Although  they  engaged  in  some  reflection  and  monitoring,  they 
lacked  clear  criteria  for  relevancy  through  out  the  project.  Therefore,  their  monitoring  attempts  resulted  in 
frustration  rather  than  progress. 

Question  3:  Evidence  for  Developing  Coherence 

The  third  question  provides  a means  for  summarizing  the  efforts  of  the  four  groups  in  terms  of  the 
integration  of  their  project  ideas  and  search  results  into  a coherent  project.  We  defined  coherence  as  the  extent  to 
which  project  methods  and  resources  were  consistent  with  stated  purposes.  We  approached  this  question  in  two 
ways.  First,  we  examined  the  protocol  data  for  instances  of  cohesion  or  processing  that  suggested  the  projects  were 
headed  toward  cohesion.  Then,  we  examined  the  project  write-ups  for  evidence  of  coherence. 

The  examination  of  the  protocol  data  revealed  two  broad  categories:  fragmentation  and  consolidation. 
Fragmentation  occurred  when  participants  failed  to  evaluate  the  usefulness  of  web  resources  in  light  of  their  project 
idea.  So,  when  participants  were  engaged  in  data-driven  searches,  fragmentation  was  common.  As  the  project 
developed,  and  searching  became  more  goal  driven,  most  participants  consolidated  their  ideas  and  web  resources 
into  a functionally  bound  instructional  activity.  This  process  involved  a discarding  and  evolving  of  ideas  that  could 
be  actualized  using  identified  web  resources.  Participants  went  through  several  iterations  of  integrating  ideas  and 
accessed  web  sites  in  an  attempt  to  stabilize  the  overall  project  idea.  The  process  required  them  to  step  back  and 
integrate  the  activity  that  will  be  posed  to  their  students,  clarify  the  reasons  for  their  decisions,  evaluate  them  in  light 
of  project  requirements,  and  determine  how  use  of  the  web  fits  into  the  process.  This  was  a difficult  process  for  all 
cases  and,  as  a result,  they  all  occasionally  experienced  frustration  and  confusion.  Although  all  cases  achieved  some 
level  of  consolidation,  both  cases  3 and  4 were  less  successful  than  1 and  2,  and  showed  more  evidence  of 
fragmentation  than  coherency. 

Persistent  Fragmentation.  Janie  (Case  3)  began  with  an  idea  about  having  students  (grades  2 or  3)  research 
local  prices  of  groceries  and  compare  them  with  prices  in  different  areas  of  the  country  (later  she  expanded  to  other 
countries,  but  went  back  to  just  the  US  in  her  final  project).  As  she  began  exploring  the  web,  she  realized  she  could 
structure  the  activity  in  several  ways:  (1)  by  asking  students  to  use  on-line  advertisements  for  grocery  prices  for 
different  areas  of  the  country;  (2)  by  having  students  communicate  via  e-mail  with  other  students  in  different  parts  of 
the  country  regarding  grocery  prices;  and  (3)  by  using  tools  such  as  cost  of  living  comparisons  to  help  explain  the 
reasons  for  the  differences.  While  using  the  Web,  she  focused  almost  exclusively  on  one  alternative  at  a time: 
finding  sites  for  collaboration  with  schools  or  finding  sites  for  cost  of  living  comparisons  or  finding  on-line  grocery 
sites.  Fitting  the  pieces  together  was  challenging  for  Janie,  and  she  seldom  stepped  back  to  grapple  with  the  whole 
picture. 
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Janie’s  write-up  of  her  project  reflected  her  fragmentation.  She  first  stated  that  the  objective  of  her  project 
was  for  students  to  find  out  the  cost  of  foods  around  the  country  and  why  the  costs  are  so  different.  However,  she 
later  stated,  ‘‘I  really  don’t  know  how  to  give  them  an  answer  to  “why?”  but  onlyto  allow  them  to  understand  and 
realize  that  not  all  things  are  the  same  as  where  they  live.”  In  other  words,  she  never  quite  integrated  her  project 
ideas.  Her  Web  resources  included:  (a)  e-mail  contacts  with  classes  around  the  country;  (b)  sites  for  looking  up 
prices  for  favorite  foods  around  the  country;  and  (c)  several  sites  for  cost  of  living  comparisons.  The  relationships 
between  these  three  resources  were  never  made  clear. 

Case  4 also  demonstrated  many  instances  of  fragmentation.  Desi  et  al.  spent  most  of  their  time  inefficiently 
searching  for  sites  on  the  art  and  history  associated  with  the  time  period  when  Shakespeare  was  writing.  When  they 
found  a site  on  Renaissance  Fair  Garb,  Nell  noted,  “Perfect.  So  each  group  is  going  to  specialize  in  an  area.  They 
will  each  have  to  come  up  with  a costume.  Everyone  will  dress  up  and  bring  food...”  Desi  replied,  “Now  I’m 
confused.  I thought  it  was  going  to  be  about  what  would  make  Shakespeare  more  understandable  to  them.”  Nell, 
‘Svell,  they  will  have  to  study  a passage  anyway,  so  have  them  perform  it.  They  will  already  be  dressed  up.” 

After  two  days  of  searching  for  world  events  and  art  associated  with  Shakespeare’s  time,  Nell  noted,  “Why 
don’t  we  just  [change]  our  project.  They  don’t  necessarily  need  to  know  the  art.  Art  had  no  influence  on  history.  Art 
didn’t  really  influence  Shakespeare.”  Desi  interrupted,  and  requested  that  they  change  the  focus,  “[our  students]  will 
have  to  read  Beowulf.  So,  let’s  go  for  it.”  Opal  asked,  “So  what  is  our  point  with  Beowulf?”  Desi  replied,  “we  want 
them  to  understand  the  culture.  It’s  a hunter  and  gatherer  kind  of  society.”  Nell  said,  “We  can  just  change  everything 
we  have  for  Shakespeare  and  apply  it  to  Beowulf.  To  understand  it,  you  also  have  to  have  an  overall  comprehension 
of  the  culture.”  After  further  discussion.  Opal  finally  suggested  a move  toward  consolidation,  “let’ s just  scrap  all  of 
that  and  focus  on  the  comprehension  of  the  play  itself.  We  found  all  of  those  sites  that  break  down  the  language  and 
they  can  go  into  those  chatrooms  and  discuss  with  those  other  people  the  questions  they  have  about  the  play,  and  see 
if  they  can  find  out  through  those  rooms  and  e-mails,  the  perspectives  of  other  people  and  their  interpretations  of  the 
p|ay.” 

Unfortunately,  Case  4 had  only  rare  moments  when  they  moved  toward  consolidation.  Their  project  write- 
up demonstrated  considerable  fragmentation.  They  stated  that  their  objective  was  to  make  learning  of  Shakespeare 
more  meaningful  than  the  traditional  teacher-directed  approach.  In  examining  their  project,  it  became  evident  that 
their  main  struggle  throughout  this  study  was  not  knowing  how  to  operationalize  that  goal  using  Web  resources. 

They  ended  up  including  three  categories  of  Web  resources:  (a)  numerous  sites  about  Shakespeare’s  life  and  times; 
(b)  sites  for  criticism;  (c)  sites  for  chat  rooms  about  his  works.  The  main  requirement  for  their  students  would  be  to 
develop  a Web  page  that  included  excerpts  from  an  (unspecified)  play  translated  into  modem  language  along  and 
other  information  of  their  choice  about  Shakespeare.  The  purpose  of  each  of  the  tasks  implied  by  the  resources  was 
not  clear,  nor  were  the  relationships  among  the  resources  and  the  tasks  made  clear.  Additionally,  two  of  the  three 
resource  categories  were  for  information  on  Shakespeare  that  could  be  found  in  one  or  two  books. 

Successful  Consolidation  and  Integration  of  Project  ideas.  Project  Requirements,  and  Information  Sources. 
Unlike  Cases  3 and  4,  Case  I (Jack,  Jane,  & Jill)  demonstrated  considerable  consolidation  early  on  in  the  work  on 
their  project.  They  chose  a travel-related  topic  that  would  require  students  to  use  the  Web  primarily  to  gather 
extensive  information  to  plan  a vacation  itinerary.  Once  they  identified  a specific  sub-topic  (e.g.,  car  rental  rates), 
they  searched  for  Web  sites  consistent  with  the  topic,  then  noted  the  URL.  When  consolidating  ideas  and  available 
resources,  they  often  discounted  options  or  simplified  the  structure  and  scope  of  their  project  to  keep  it  manageable. 
For  instance,  when  grappling  with  the  question  of  student  autonomy  in  choosing  vacation  destinations,  Jill  said  "so 
do  we  want  to  leave  it  completely  up  to  them,  or  give  them  a scenario  such  as  you're  going  to  California?"  Jack 
replied,  "It  might  be  easier  to  say  you're  going  to  California."  Jill  went  on  to  explain,  "it  would  be  easier  on  them,  but 
also  on  us,  because  we  would  know  what  were  are  looking  for  in  grading  this."  She  explained  to  one  of  the 
instructors  that  "verifying"  students'  work  would  require  such  structure.  Although  the  group  failed  to  consider  varied 
notions  of  teaching  that  would  entail  alternative  evaluation  techniques  in  this  instance,  they  did  move  toward  further 
consolidation. 

Their  project  write-up  also  showed  evidence  of  consolidation.  They  wrote  that  the  purpose  of  their  project 
was  to  involve  students  in  an  authentic  learning  task  that  would  require  them  to  integrate  their  knowledge  of 
geography,  math,  language  arts,  etc.  to  plan  a vacation  using  the  Internet.  Students  would  choose  from  three 
vacation  planing  scenarios  that  stipulated  the  state,  the  total  budget,  time  frame,  and  number  of  family  members. 
Students  would  have  to  use  those  constraints  to  plan:  how  they  will  get  there;  where  they  will  stay;  what/where  they 
will  eat;  how  they  will  get  around;  and  what  they  will  do  for  entertainment.  They  identified  three  categories  of  web 
sites  for  their  students  to  use:  (a)  mapping;  (b)  travel  agencies;  and  (c)  state  tourist  information.  They  clearly 
articulated  how  these  categories  would  be  needed.  The  plan  was  to  help  their  students  get  started  with  a few  good 
sites,  then  turn  it  over  to  them. 

The  extent  to  which  participants  were  successful  consolidating  the  web  resources  into  a coherent  project 
was  largely  dependent  on  whether  or  not  their  metacognitive  processing  involved  articulation  of  a project  goal  and 
monitoring  of  search  progress  in  light  of  that  goal.  Although  Tammy  and  Kara  (Case  2)  took  longer  than  Jack  et  al. 
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(Case  1)  to  articulate  their  final  project  goal,  they  were  just  as  successful  in  developing  a coherent  project.  Case  2 
needed  to  engage  in  more  negotiation  of  what  the  project  would  entail  since  Kara  was  inclined  toward  direct  teaching 
methods  while  Tammy  was  in  favor  of  creating  a more  student-  directed,  open  learning  environment.  This 
negotiation  process  seemed  to  encourage  their  consolidation  processing. 

The  written  report  developed  by  Case  2 provided  strong  evidence  for  a coherent  project.  Interestingly, 
Tammy  and  Kara  included  the  fewest  number  of  web  resources.  In  addition  to  the  EduStock  site,  they  anticipated 
their  students  would  want  to  access  the  financial  profiles  of  favorite  companies  and  might  want  to  explore  an  on-line 
Investment  Club  to  get  a feel  for  the  language  used  by  investors.  Much  of  the  information  their  students  would  need 
was  available  within  the  EduStock  site.  It  provided  some  general  information  on  the  stock  market,  allowed  for 
monitoring  the  performance  of  companies  on  the  stock  market,  and  included  a program  for  simulating  investment  in 
companies  one  chooses.  Although  Case  2*s  project  would  require  the  least  amount  of  web  searching  for  future 
students,  relative  to  the  projects  of  the  other  three  cases,  it  clearly  would  require  the  highest  level  of  cognitive  effort 
and  complex  problem  solving  on  the  part  of  their  future  students. 

Conclusions 

The  study  uncovered  several  prominent  findings  with  implications  for  understanding  how  learners  generate 
and  evolve  project  ideas  while  using  the  Web.  All  students  were  able  to  generate  a project  in  an  open-ended, 
problem-based  environment  using  an  information  technology.  Although  they  experienced  some  difficulty  and 
frustration  during  the  process,  they  nonetheless  all  achieved  adequate  success  to  meet  their  course  requirements  and 
they  all  developed  in  their  sophistication  of  using  the  Web.  Yet,  three  findings  are  particularly  noteworthy. 

First,  learners  needed  to  move  beyond  data-driven  processes  and  become  more  goal  driven  in  order  to 
efficiently  progress  toward  a coherent  project  idea.  All  groups  experienced  difficulty  in  the  process  of  consolidating 
their  web  project  ideas  with  class  project  requirements  and  with  the  vast  number  of  accessed  web  resources  and 
tools.  However,  learners  who  identified  specific  goals  and  project  ideas  up  front,  and  then  used  the  web  to  find 
resources  consistent  with  the  identified  activity,  achieved  more  coherence.  Participants  whose  final  projects  did  not 
reflect  substantial  coherence  tended  to  rely  on  data  driven  strategies  and  rarely  evaluated  how  web  resources  fit  into 
their  project  idea.  The  protocols  from  the  two  more  successful  cases  showed  constant  monitoring  of  the  fit  between 
resources  located  and  the  intent  of  the  project.  Even  while  the  intent  of  the  project  was  still  evolving.  Case  1 and  2 
participants  were  simultaneously  evaluating  the  web  resources  and  the  current  project  idea. 

Second,  all  learners  went  through  several  iterations  of  integrating  ideas  and  accessed  web  sites  in  an  attempt 
to  stabilize  the  overall  project  idea.  The  process  required  learners  to  step  back  and  integrate  the  activity  that  will  be 
posed  to  their  students,  clarify  the  reasons  for  their  decisions,  evaluate  them  in  light  of  project  requirements,  and 
determine  how  use  of  the  web  fits  into  the  process.  This  was  a difficult  process  for  learners,  and  they  occasionally 
experienced  frustration  and  confusion  as  a result.  This  is  at  least  in  part  due  to  the  fact  that  most  students  had 
difficulty  recognizing  limitations  in  their  thinking  about  how  to  use  and  conceptualize  the  Web  as  a tool  for  learning. 
In  addition,  many  students  seemed  motivated  to  protect  their  initial  conceptions  of  both  learning  and  the  Web, 
despite  ongoing  evidence  that  their  searching  was  not  leading  toward  clarification  of  their  projects. 

Third,  all  three  sources  of  knowledge  were  essential  for  project  development  that  was  both  efficient  and  free 
of  frustration.  Metacognitive  knowledge  was  especially  important  for  consolidation  processing  and  often  seemed 
dependent  on  the  system  and  domain  knowledge.  Only  one  case  seemed  to  be  able  to  compensate  for  a lack  of 
system  and  domain  knowledge  through  their  use  of  metacognitive  knowledge.  This  implies  that  increased 
instructional  scaffolding  will  be  necessary  for  students  using  the  Web  in  situations  where  they  are  novices  in  regard 
to  the  system  and  the  domain  being  studied.  Future  research  should  examine  how  instructional  scaffolding  might 
encourage  more  goal  driven  and  metacognitive  processing  in  open  learning  environments  that  involve  the  Web.  Our 
results  suggest  that  such  scaffolding  may  enable  students  to  achieve  greater  coherency  and  experience  less 
frustration. 
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Abstract 

To  fully  exploit  the  power  and  potential  of  web-based  instruction,  on-line  course  developers  must  confront 
and  creatively  deal  with  the  limits  imposed  by  bandwidth  capacity.  Increasing  the  rate  of  data  delivery  will 
most  certainly  become  a technological  possibility,  however,  the  media-rich  software  environments  offered 
via  the  Internet  can  force  end-users  to  consume  bandwidth  at  rates  that  surpass  its  availability.  To  deal 
with  this  challenge,  a series  of  design  strategies  are  presented  that  allow  instructional  developers  to 
maximize  the  effectiveness  of  online  materials  while  efficiently  utilizing  available  network  resources. 

The  movement  of  distance  education  to  the  desktop  computer  reflects  the  growing  demand  for  instruction 
that  is  time  and  place  independent.  The  convergence  of  audio,  video,  text  and  graphics  in  a networked,  readily 
accessible  virtual  space  has  prompted  a variety  of  instructional  packaging  solutions,  from  asynchronous  text-based 
interactions  to  real-time  desktop  videoconferencing,  enhanced  by  a never-ending  array  of  peripherals  such  as 
electronic  whiteboards  and  application  sharing  capabilities.  While  room-based  videoconferencing  also  continues  to 
grow  in  popularity,  web-based  instruction  extends  the  reach  of  traditional  distance  education  to  locations  and  hours 
beyond  such  systems.  Learners  with  an  Internet  connection  can  participate  in  coursework  at  their  convenience 
without  having  to  go  to  a given  location  at  a designated  time,  the  constraints  imposed  by  two-way  large  group 
conferencing  (Moore  & Lockee,  1999b). 

While  networked  instruction  to  the  desktop  offers  a variety  of  possibilities  for  course  delivery,  it  is  not 
without  flaw.  Slow  download  time,  delayed  audio  and  jerky  video  images  are  the  result  of  such  a system's  primary 
limitation— bandwidth.  As  distance  educators  seek  to  leverage  the  power  of  desktop  computing  for  distributed 
learning;  bandwidth  places  constraints  on  potentialities  by  placing  a limit  on  the  information  transfer  capabilities  of 
various  teaching  technologies.  Providers  encounter  challenges  maximizing  the  instructional  potential  of  this  confined 
resource.  Unfortunately,  the  task  is  not  always  easy;  as  distance  educators  often  inherit  a media  infrastructure,  and 
typically  are  not  in  a position  to  increase  the  physical  capability  of  their  systems.  Instead  they  must  choose 
appropriate  instructional  and  media  attributes  that  are  effective  and  yet  use  bandwidth  wisely.  Different  instructional 
strategies  require  different  levels  of  bandwidth.  The  physical  infrastructure  of  available  bandwidth  places  constraints 
on  the  possibilities,  but  a course  designer  has  complete  flexibility  working  under  that  ceiling(Lockee,  Moore,  & 
Moore,  in  press;  Moore  & Lockee,  1999a;  Moore  & Lockee,  1999b). 

While  networked  instruction  to  the  desktop  offers  a variety  of  possibilities  for  training  delivery,  it  is  not 
without  flaw.  Slow  download  time,  delayed  audio  and  jerky  video  images  are  the  result  of  such  a system’s  primary 
limitation — bandwidth.  To  put  it  simply,  bandwidth  places  a limit  on  what  can  occur  at  any  particular  time 
throughout  the  instructional  event.  As  the  demands  on  local  and  global  networks  soar,  so  will  the  quest  for  solutions 
to  bandwidth  constraints. 

The  Interaction  of  Bandwidth  and  Instruction 

Bandwidth  is  a volume  measure  of  information  flow.  On  a digital  network  it  is  the  number  of  bits  that  can 
be  transported  in  a given  period  of  time.  Digital  networks  are  composed  of  a number  of  components  that  may  include 
a sender’s  computer,  a connection  from  the  sender’s  computer  to  a router(s),  a connection  from  the  router(s)  to  the 
receiver’s  computer,  and  finally,  the  receiver’s  computer.  The  slowest  link  in  that  path  will  define  the  system’s 
available  bandwidth.  When  planning  network  use,  one  should  keep  in  mind  that  effective  bandwidth  is  determined  by 
the  weakest  link  in  a system  (Moore  & Lockee,  1999b). 

Instructional  design  theories  recommend  that  instruction  be  designed  before  media  is  chosen  (Smith  & 
Ragan,  1999).  Unfortunately,  in  the  practice  of  computer-based  training  programs  such  procedures  are  rarely 
possible.  In  many  organizations,  the  pervasiveness  of  the  Internet  has  made  the  web  a default  delivery  media. 
Currently,  distance  education  course  developers  must  have  knowledge  of  the  network  over  which  instruction  will  be 
delivered  and  its  capabilities  and  limitations  in  order  to  make  important  instructional  development  decisions. 

Whether  creating  instruction  for  delivery  over  a privately  owned  intranet  or  via  the  globally  accessible  Internet,  the 


instructional  designer  must  have  knowledge  of  strategies,  trends,  and  trade-offs  in  bandwidth  consumption  is 
essential  to  produce  courseware  that  is  both  technically  and  instructionally  effective  (Moore  & Lockee,  1999a). 

Bandwidth  Management  Strategies 

Bandwidth  brings  logistics  into  instructional  design  simply  because  successful  instruction  may  be 
compromised  if  instructional  materials  are  delivered  at  a rate  at  which  learners  are  incapable  of  consuming  them. 
Learner  attention,  motivation,  and  comprehension  may  suffer  when  participants  are  forced  to  continually  wait  for  an 
instructional  event  to  occur.  Fortunately,  instructional  developers  have  a number  of  strategies  for  eliminating  or 
reducing  these  delays.  These  strategies  can  be  organized  into  the  following  categories.  Pre-distribution  strategies 
attempt  to  overcome  delays  by  delivering  instructional  materials  and  images  in  advance  of  the  instructional  event. 
Parsing  strategies  emphasize  designing  materials  according  to  their  instructional  essence  in  order  to  remove 
extraneous  information  and  data.  Choreographic  strategies  attempt  to  present  instructional  materials  at  a rate  equal 
to  the  rate  at  which  they  are  consumed  (Lockee  et  al.,  in  press). 

Pre-distribution  strategies 

Pre-distribution  strategies  are  techniques  that  assist  in  the  avoidance  of  bandwidth  bottlenecks  by 
distributing  data  through  other  means.  Strategic  use  of  alternative  media  such  as  print  materials,  CD-ROMs  and 
videotapes  can  dramatically  increase  the  volume  of  instructional  materials  to  which  learners  have  access  while 
reducing  demands  on  the  delivery  network.  This  technique  has  been  called  a “hybrid  approach”  (Douglas,  1993)  to 
distance  education  because  it  incorporates  the  instructional  strengths  of  a variety  of  teaching  technologies.  However, 
traditional  distribution  routes  are  required  for  this  strategy  to  have  a successful  impact.  Also,  these  tools  have  the 
disadvantage  of  having  to  be  developed  or  purchased  well  in  advance  of  the  instructional  event.  To  produce  these 
materials  a number  of  skills  and  equipment  may  be  required.  Perhaps  most  important  in  using  these  tools  is  the 
ability  to  evaluate  commercial  or  previously  created  instructional  materials. 

Pre-distribution  strategies  have  the  disadvantage  of  increasing  the  planning  time  of  an  online  course  in  order 
to  carry  out  the  logistics  of  traditional  materials  delivery  (library  checkout,  postal  service,  etc.).  However,  pre- 
distribution has  the  advantage  of  encouraging  pre-planning  and  creating  contingency  plans. 

Parsing  strategies 

A brief  look  at  many  web-based  training  sites  reveals  that  a large  percentage  of  the  web  site  components  are 
extraneous  to  instructional  goals,  such  as  the  inclusion  of  images  for  aesthetic  purposes.  By  eliminating  web  site 
content  that  is  aesthetic  but  not  instructional,  relevant  data  throughput  can  be  maximized. 

The  next  step  is  to  reduce  unnecessary  data  within  instructional  content.  Training  developers  must  ask 
themselves  what  media  attributes  are  essential  to  the  facilitation  of  learning.  For  example,  is  animation  needed  to 
convey  a process  or  will  a series  of  still  photographs  serve  the  same  purpose?  The  advantages  of  doing  so  are 
functional  as  well  as  monetary.  Providing  information  through  text  instead  of  a video  clip  of  a talking  head  is  equally 
effective  and  consumes  only  a small  portion  of  bandwidth  compared  to  digital  video. 

Choreography  strategies 

The  final  web-based  delivery  strategy  is  called  choreographic  because  it  deals  with  the  order  in  which 
training  is  delivered.  Images  and  other  media  that  are  data  intensive  can  be  strategically  interspersed  with  text.  A 
large  volume  of  text  can  be  transmitted  relative  to  images.  In  terms  of  data,  text  delivers  more  content  than  an 
equivalent  amount  of  data  in  an  image.  While  students  are  reading  textual  material  images  can  be  downloaded  and  be 
ready  to  be  examined  once  the  text  has  been  consumed.  This  strategy  may  make  the  educational  experience  seamless 
and  the  delays  imposed  by  bandwidth  may  not  be  noticed.  A similar  strategy  is  often  used  at  amusement  parks.  Many 
rides  require  a long  wait.  Many  parks  are  now  using  this  wait  time  to  prepare  the  audience  for  the  experience  by 
showing  background  videos  and  other  contextual  information.  The  same  can  be  done  in  an  online  environment. 
Secondary  presentation  forms  that  augment  the  instructional  message  can  be  included  when  a data  heavy  image  is 
required. 


Conclusion 

In  a distributed  learning  environment,  the  delivery  network  infrastructure  must  be  considered  so  that 
instructional  developers  can  avoid  creating  instruction  that  diverts  a learner’s  attention  because  of  unnecessary 
delays  (Moore  & Lockee,  1999a).  While  the  above  strategies  can  be  implemented  to  utilize  bandwidth  more 
efficiently,  it  is  important  to  remember  a point  well-made  by  (Cohen  & Rustad,  1998),  that  “technology  should 
function  as  a tool  for  learning,  not  as  the  blueprint  that  drives  the  learning  agenda”  (1998,  p.  32).  The  wise  use  of 
these  bandwidth  strategies  increases  control  over  the  learning  event  while  reducing  the  influence  the  delivery  media 
has  on  instructional  goals  and  objectives. 
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Abstract 

Recent  exponential  growth  in  the  field  of  distance  education  has  unfortunately  not  been  matched  with  equal 
growth  in  a quality  research  base  in  terms  of  informing  effective  practice.  In  their  1996  analysis  of  the 
distance  education  literature , Mclsaac  & Gunawardena  state  that  "much  research  has  taken  the  form  of 
program  evaluation , descriptions  of  individual  programs,  brief  case  studies,  institutional  surveys  and 
speculative  reports " (1996,  p.  421).  In  the  past  few  years,  a particular  form  of  study  regarding  the 
effectiveness  of  distance  learning  has  enjoyed  increased  publication.  Instructional  technology  professionals 
will  recognize  the  methodology  behind  these  reports  as  the  media  comparison  study,  newly  revived  for  use 
in  a distance  education  setting.  Media  comparison  studies  have  historically  formed  the  basis  of  much 
research  in  distance  education  ( Mclsaac  & Gunawardena,  1996;  Schlosser  & Anderson,  1994),  but  are 
lately  becoming  even  more  common.  These  studies  are  predictably  plagued  with  the  same  design  issues  as 
their  predecessors,  however  their  "no  difference"  outcomes  are  being  reported  for  politically  different 
reasons.  This  paper  details  the  origins  of  the  media  comparison  study,  its  current  use  as  an  evaluation 
instrument  in  distance  education,  and  recommendations  for  more  stringent  discrimination  between 
research  and  evaluation  in  the  field  of  distance  learning. 

History  of  research  in  instructional  technology 

Since  the  adoption  of  modem  media  for  instructional  purposes,  innumerable  attempts  have  been  made  to 
measure  the  effect  that  a given  technology  has  on  student  achievement.  Early  in  the  history  of  electronic  technologies 
like  film  and  radio,  educational  researchers  were  driven  to  demonstrate  that  these  revolutionary  devices  had  a 
positive  impact  on  learning  (Saettler,  1968).  The  most  common  approach  to  attempt  this  investigation  was  the 
“media  comparison”  study,  so  named  because  of  its  strategy  of  comparing  the  learning  outcomes  of  an  experimental 
group  receiving  instructional  content  via  one  medium  against  the  outcomes  of  a control  group  receiving  the  same 
content  through  a different  medium.  The  most  popular  control  group  was  the  “traditional”,  or  lecture  format  class, 
with  the  instructor  serving  as  the  delivery  medium.  Even  though  comparison  studies  were  fairly  simple  in  design, 
variations  of  the  research  strategy  exist.  Most  maintained  the  “media”  as  the  independent  variable,  but  some  used  the 
same  instructional  method  (e.g.,  presentation  via  live  lecture  vs.  presentation  via  videotape  of  the  same  lecture)  while 
others  utilized  different  instructional  methods  and  different  media  (presentation  via  lecture  vs.  problem  solving  via 
computer-based  instruction)  (Ross  & Morrison,  1996). 

The  use  of  media  comparison  studies  dates  back  to  the  origins  of  mediated  instruction.  Early  researchers  in 
audiovisual  education  worked  diligently  to  control  all  aspects  of  such  experiments  so  that  results  would  maintain 
validity  and  comparisons  would  be  fair.  For  example,  McClusky  & McClusky  produced  a “Comparison  of  six 
modes  of  presentation  of  subject  matter”.  Two  trial  experiments  were  conducted  comparing  six  methods  of 
presentation  for  the  content  depicted  in  two  separate  lessons:  film  only,  slides  with  subtitles  only,  photographic 
pictures  with  subtitles  only,  and  each  medium  with  a supplemental  question  and  answer  session.  Experiments  were 
carefully  controlled  so  participants  viewed  slide  and  print  images  for  exactly  the  same  time  as  such  images  appeared 
in  the  film.  Recall  of  content  was  measured  for  each  group  using  multiple-choice  tests.  The  outcomes  reflect  some  of 
the  earliest  evidence  of  what  Russell  (1997)  calls  the  non-significant  difference  (NSD)  phenomenon:  ‘These 
comparisons  show  such  inconsistent  results  that  the  film,  slide,  and  print  appear  to  possess  no  distinct  advantage  one 
over  the  other  as  far  as  these  particular  experiments  are  concerned”  (Russell,  1997,p.  257).  And,  as  history  repeats 
itself,  the  tendency  was  to  try  this  research  design  with  the  advent  of  any  newer  technological  innovation,  with 
consistent  production  of  the  same  non-significant  results. 

In  1983,  Richard  Clark,  one  of  the  most  renowned  critics  of  instructional  technology  research,  detailed  the 
problems  inherent  in  media  comparison  studies  and  the  improper  assumptions  about  their  outcomes.  He  stated  that 
“these  findings  were  incorrectly  offered  as  evidence  that  different  media  were  ‘equally  effective’  as  conventional 
means  in  promoting  learning.  No  significant  difference  results  simply  suggest  that  changes  in  outcome  scores  did  not 
result  from  any  systematic  differences  in  the  treatments  compared”  (Clark,  1983,  p.  447).  Clark  emphasized  that 
media  are  merely  the  delivery  mechanisms  for  instructional  content  and  do  not  impact  the  learning  process.  This 
perspective  spurred  great  debate  within  the  field  of  instructional  technology  (Clark,  1994a;  1994b;  Kozma,  1991; 
1994a;  1994b;.  Jonassen,  Campbell,  & Davidson,  1994;  Morrison,  1994;  Reiser,  1994;  Shrock,  1994;  Tennyson, 
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1994).  While  some  contend  that  certain  attributes  of  media  can  and  do  effect  learning  outcomes  (Kozma,  1991; 
1994a,  1994b),  Clark  maintains  that  it  is  instructional  method  that  influences  learning,  not  the  delivery  medium 
(Clark,  1983;  1994a;  1994b).  While  this  debate  will  undoubtedly  continue,  the  futility  of  comparison  studies  to 
measure  the  impact  of  media  on  learning  is  consistently  recognized  in  the  field  of  instructional  technology  (Ross  & 
Morrison,  1996).  A chronological  collection  of  hundreds  of  such  experiments  can  be  found  at 
http://www2.ncsu.edu/oit/nsdsplit.htm  (Russell,  1997).  The  database  serves  as  a reminder  to  researchers  that 
comparative  designs  will  continue  to  provide  predictable,  non-significant  outcomes. 

While  the  tendency  is  to  compare  learning  outcomes  via  different  media  is  to  demonstrate  the  greater 
effectiveness  of  the  newer  medium,  distance  education  comparison  studies  have  given  the  argument  a new  twist.  As 
evidenced  in  the  following  examples,  the  outcomes  are  now  used  to  demonstrate  that  the  distance-delivered 
instructional  event  is  at  least  equal  to  the  campus-based,  face-to-face  version.  Kanner,  Runyon,  & Desiderato  (1958) 
espoused  this  approach  in  less  optimistic  terms  when  summarizing  their  televised  instruction  research  by  stating  that 
televised  sessions  were  no  more  detrimental  to  classroom  learning  than  face-to-face  instruction. 

Recent  research  in  distance  learning 

As  anyone  involved  in  the  support  of  distance  education  programming  is  aware,  the  resources  required  to 
deliver  such  programming  to  geographically  and  temporally  dispersed  learners  are  not  inconsequential.  Though  cost- 
saving goals  are  often  highlighted  in  plans  for  reaching  new  and  different  student  markets,  the  front-end  investments 
needed  in  course  development,  delivery  infrastructures,  teaching  technologies,  and  support  staff  can  be  formidable 
(Keegan,  1996;  Musial  & Kampmueller,  1996).  In  analyzing  the  use  of  distance  program  evaluation  data,  Thorpe 
(1988)  explains  that  administrators  venturing  into  this  new  educational  arena  are  expectedly  anxious  to  use  positive 
evaluation  results  to  promote  the  desirable  aspects  of  providing  opportunities  for  remote  student  clientele.  Increased 
access  to  such  programming  does  not  seem  to  serve  as  a satisfactory  benefit  for  the  implementation  of  distance 
education  efforts.  Stakeholders  desire  to  demonstrate  that  participants  in  distance-delivered  courses  receive  the  same 
quality  of  instruction  off-campus  as  those  involved  in  the  “traditional”  classroom  setting.  What  better  way  to 
determine  the  equality  of  experiences  than  to  compare  student  achievement  between  the  two  groups?  For  example, 
according  to  Newlands  & McLean  (1996)  “The  calming  of  fears  about  the  quality  of  distance  learning  has  been 
assisted  by  evidence  that,  in  terms  of  assessment,  distance  students  perform  as  well  as  internal  students...”  (p.  289). 
One  of  the  most  prominent  early  works  in  teleconferencing  training.  Bridging  the  Distance  (Monson,  1978),  employs 
a collection  of  comparison  studies  for  this  very  reason — to  ensure  soon-to-be  distance  educators  that  off-campus 
students  will  be  just  as  academically  successful  as  their  campus-bound  counterparts. 

The  guaranteed  validation  of  equality  of  learner  achievement  has  led  to  use  of  comparison  studies  about 
distance  education  in  almost  every  imaginable  discipline.  The  research  design  remains  exactly  the  same  as  previously 
compared  mediated  experiences,  the  on-campus  students  serve  as  the  control  group,  since  their  experience  is 
unmediated,  while  the  distant  students  provide  the  treatment  group.  For  example,  "The  38  South  Carolina  campus 
students,  considered  the  control  group  in  this  report,  completed  either  all  or  the  majority  of  their  degree  programs  in 
traditional  classroom  settings  on  the  Columbia  campus”  (Douglas,  1996,  p.  878).  Repeatedly,  outcomes  are 
embodied  in  statements  like  “there  was  no  real  difference  between  grades  of  in-class  and  ITV  students”  (Fox,  1996, 
p.  362).  Some  authors  offer  additional  analysis  such  as,  ‘There  were  no  differences  between  pre-and  post-tests 
(which  measure  increases  in  knowledge)  across  sites.  This  demonstrates  that  the  program  was  effective  in  increasing 
knowledge...”  (Reiss,  Cameon,  Matthews,  & Shenkman,  1996,  p.  350).  Behind  conclusions  such  as  these  is  the 
conviction  that  distance  learners  are  engaged  in  an  equally  rigorous  instructional  experience  even  though  they  are  not 
participating  in  campus-based  education.  General  unawareness  of  the  history  of  instructional  technology  research, 
especially  the  out-dated  notion  of  the  effectiveness  of  comparison  studies,  is  exemplified  in  a recent  sociology 
journal  through  the  following  excerpt: 

Of  greatest  concern  to  us  is  the  absence  of  well-crafted  comparison  studies,  that  examine  not  simply  student 
attitudes  towards  distance  learning,  but  the  actual  knowledge  and  skills  that  students  acquire  from  televised 
teaching.  Ideally,  this  demonstration  could  involve  the  same  instructor  teaching  two  sections  of  the  same  course 
during  the  same  school  term,  one  exclusively  by  live  instruction  and  the  other  only  by  distance  learning  (Thyer, 

Polk,  & Gaudin,  1 997,  p.  367). 

Instructional  technology  research  journals  are  not  exempt  from  the  publication  of  such  studies.  In  a 1996 
issue  of  Educational  Technology  Research  & Development , Whetzel,  Felker,  and  Williams  began  their  article  “A 
Real  World  Comparison  of  the  Effectiveness  of  Satellite  Training  and  Classroom  Training”  with  an  analysis  of 
research  regarding  the  effectiveness  of  televised  instruction,  summing  up  the  mixed  results  by  re-stating  Clark’s 
(1994)  view  that  “...any  necessary  teaching  method  can  be  delivered  by  many  media  with  similar  learning  results 
(Whetzel,  Felker,  & Williams,  1996,  p.  6).  However,  their  study  used  a research  design  that  compared  the 
achievement  of  the  on-site  versus  distant  students: 
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-For  the  two  courses  in  which  satellite  and  classroom  training  -were  compared,  an  analysis  of  covanaace 
(ANCOVA)  was  used  to  compare  delivery  modes  for  nonequivalent  groups  (satellite  and  classroom  participants), 
using.pretest  score  as.the  covariate  and  posttest  score  as  the  dependent  variable  (Whetzel  et  al.,  1996,  p.  .10). 

The  use  of  media  comparison  studies  in  distance  learning  is  not  limited  to  higher  education  settings.  Barry 
& Runyan  (1995)  assembled  a “Review  of  Distance  Education  Studies  in  the  U.S.  Military”  in  which  they  cited 
eight  “empirical  studies  that  compared  student  achievement  in  distance  learning  courses  to  achievement  in 
comparable  resident  courses”  (p.  43).  Their  closing  statement  embraced  the  reliable  non-significant  difference  results 
as  proof  that  the  U.S.  military  could  safely  continue  to  invest  in  the  expansion  of  distance  learning  initiatives.  Due  to 
the  expanded  publication  of  studies  like  these,  a distinction  must  be  made  between  valid  research  in  distance 
education  and  evaluation  efforts  for  distance  program  confirmation. 

Methodology  analysis 

Although  we  often  use  the  terms  interchangeably,  as  we  have  noted  earlier,  evaluation  and  research  are  not 
the  same  although  they  may  share  many  methods. 

Evaluation  is  practical  and  concerned  with  how  to  improve  a product  or  whether  to  buy  and  use  a product. 
Studies  that  compare  one  program  or  media  against  another  are  primarily  evaluation.  Evaluations  seeks  to  find  the 
programs  that  “work’5  more  cheaply,  efficiently,  quickly,  effectively,  etc. 

Research,  on  the  other  hand  tends  to  be  more  concerned  with  testing  theoretical  concepts  and  constructs  or 
with  attempting  to  isolate  variables  to  observe  their  contributions  to  a process  or  outcome  (Moore,  Myers,  & Burton,* 
1994,  p. 35) 

Research  studies  generate  hypotheses  from  theory.  In  the  case  of  so  called  “hard”  sciences  such  as  physics, 
these  predictions  are  usually  a quantitative  point  value,  magnitude,  or  form  function  which  become  point  predictions 
(Meehl,  1967).  Point  predictions  become  easier  to  refute  as  measurement  improves,  that  is,  the  better  the 
measurement  the  more  the  hypothesis  is  exposed  to  rejection.  (Indeed  many  replications  involve  changes  in  measures 
rather  than  “study”  conditions).  Rejection,  in  the  case  of  a point  prediction,  is  a modus  tollens  refutation  (i.e.  T ->  E, 
not  E,  therefore  not  T)  (Popper,  1968).  Research  in  the  “soft”  sciences  however,  does  not,  indeed  cannot,  test  point 
predictions.  Rather,  the  logical  compliment  of  the  predicted  outcome,  the  point-null  hypothesis  (there  is  no 
difference  between,  for  example,  two  groups  of  participant’s  mean  scores)  is  tested.  Interestingly,  the  effect  of  this 
manipulation  is  that  the  theoretically  derived  hypothesis  is  not  subjected  to  true  modus  tollens  - it  cannot  be 
logically  refuted  (Meehl,  1967).  The  null  hypothesis  can  be  accepted  but  not  embraced  as  true,  yet  it’s  acceptance 
does  not  refute  the  core  hypothesis  (Orey,  Garrison,  & Burton,  1989).  For  this  reason  (among  others  such  as  the 
“loose”  connection  between  theory  and  variables),  it  is  often  the  case  that  “soft”  science  researchers  commonly 
speculate  about  what  could  have  occurred  when  the  null  is  accepted.  Such  post  hoc  speculation  is  permissible 
because  the  core  hypothesis  cannot  be  truly  refuted.  Such  theories  never  actually  die,  researchers  just  sort  of  “.  . .lose 
interest  in  the  thing  and  pursue  other  endeavors”  (Meehl,  1978,  p.  807).  Theory-based  research  studies  are  not  good 
candidates  to  be  “repurposed”  as  “negative”  evidence  that  something  didn’t  happen. 

A “second  level”  problem  relates  to  what  Reeves  (1993,  1995)  among  others  has  referred  to  as 
pseudoscience.  Reeves  offers  nine  characteristics  of  pseudoscientific  studies  and  estimates  that  perhaps  60%  - 70% 
of  “empirical-quantitative”  studies  in  the  major  instructional  technology  research  journals  suffer  from  two  or  more  of 
these  flaws.  The  bulk  of  these  weaknesses  such  as  failing  to  link  the  study  to  a robust  theory,  poor  literature  review, 
weak  treatment  implementation,  measurement  flaws,  inadequate  sample  size,  and  poor  analyses  (Reeves,  1993; 

1995))  bias  the  research  towards  not  finding  a statistically  significant  difference  (Burton  & Magliaro,  1988).  In  other 
words,  bad  science  and  bad  designs  can  produce  no  differences. 

The  last  two  comments  relate  to  both  theoretical  and  evaluation  comparative  studies.  The  first  comment  is 
that  when  we  test  a hypothesis,  we  test  not  just  the  variable  of  interest,  we  also  test  the  assumption  of  ceteris  paribus 
(all  things  are  assumed  to  be  equal  except  for  those  conditions  that  are  actually  manipulated)  (Orey  et  al.,  1989).  It  is 
in  a fact  a ‘folk”  version  of  ceteris  paribus  which  researchers  often  resort  to  when  they  explain  the  failure  to  find  a 
predicted  statistically  significant  difference  by  resorting  to  differences  in  the  sample  or  task  that  were  outside  of  what 
was  being  manipulated.  To  the  extent  that  ceteris  paribus  is  not  true,  the  results  of  the  study  (in  either  “direction”) 
are  suspect.  With  test  such  as  ANOVA  and  ANCOVA,  this  relates  to  the  assumption  of  homogeneity  of  variance. 
This  assumption  is  often  not  tested  because  the  tests  are  assumed  to  be  robust  to  such  violations  (Thompson,  1993). 
Unfortunately,  it  does  not  appear  to  be  true  with  ANCOVA  (see,  e.g.  Keppel  & Zedeck,  1989)  and  may  not  be  true 
of  ANOVA  either  (e.g.  Wilcox,  Charlin,  & Thompson,  1986). 

Second,  in  research  or  evaluation,  measurement  is  a problem.  Faulty  measurement  was  another  of  Reeves’ 
(1993,  1995)  indicators  of  pseudoscience  but  in  evaluation,  particularly  as  it  relates  to  “real  world”  educational 
contexts  such  as  found  in  distance  or  distributed  education,  the  problem  is  often  more  insidious.  The  burden  of 
showing  reliability  and  validity  for  any  test  not  in  general  use  is  always  upon  the  researcher  (Burton  & Magliaro, 
1988).  Many  studies  related  to  distance  learning  use  “teacher-made  achievement  tests”  which  may,  or  may  not  have 
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reliabilities  or  validaties  established.  Perhaps  worse  than  using  a test  which  produces  scores  which  are  largely  error 
or  unrelated  to  the  content  however,  is  the  fact  that  such  tests  are  often  used  a part  of  a graded  exercise.  Graded 
exercises  may  cause  people  who  tend  to  make  A’s  and  B’s  simply  work  harder  to  overcome  any  problem  in  the 
instruction.  The  potential  lack  of  adequate  tests  are  measurement  problems.  The  potential  difference  in  effort  are 
violations  of  ceteris  paribus. 

In  terms  of  statistics,  many  current  researchers  have  argued  that  null  hypothesis  testing  should  be  eliminated 
altogether  (e.g.  Carver,  1993)  while  others  such  as  Thompson  (1996)  and  Robinson  & Levin  (1997)  would  like  to 
see  such  tests  supplemented.  Although  there  are  differences,  both  camps  tend  to  agree  on  two  things:  effect  size  and 
replication.  Effect  sizes  should  always  be  reported  according  to  Thompson  (1996)  but,  as  Levin  (1993)  points  out  “to 
talk  of  effect  sizes  in  the  face  of  results  that  are  not  statistically  significant  does  not  make  sense”  (p.  379) 

Replication  refers  to  repeating  essentially  the  same  experiment  multiple  times.  No  finding  should  ever  stand 
on  a single  study.  It  is  worth  noting  however,  that  while  some  believe  that  such  experiments  can  inform  social 
science  theory  (e.g.  Phillips,  1992),  others  (e.g.  Salomon,  1991)  believe  that  no  matter  how  well  constructed 
experimental  and  similar  research  approaches  are,  "they  are  based  on  a number  of  assumptions,  none  of  which  fit  the 
study  of  whole  classroom  cultures”  (p.13).  We  assume  this  would  include  distant  classrooms  and  distributed  cultures. 

Finally  we  offer  the  following  caveat  related  to  accepting  NSD  studies  as  proof.  Establishing  a null  is  very 
much  like  the  not  guilty  assumption  of  the  US  legal  system.  In  both  cases,  the  burden  of  proof  is  on  overturning  the 
assumption  based  on  evidence.  But  failure  to  reject  the  null  hypothesis  means  just  that  and  nothing  more;  just  as  a 
legal  finding  of  not  guilty  does  not  mean  innocent.  As  Carver  (1978)  puts  it: 

What  is  the  probability  of  obtaining  a dead  person  given  that  the  person  was  hanged?  Obviously,  it  is  very 
high,  perhaps  .97  or  higher.  Now,  let  us  reverse  the  question.  What  is  the  probability  that  a person  has  been  hanged, 
given  that  the  person  is  dead?  This  time  the  probability  will  undoubtedly  be  very  low,  perhaps  .01  or  lower.  No  one 
would  be  likely  to  make  the  mistake  of  substituting  the  first  estimate  (.97)  for  the  second  (.01);  that  is  to  accept  (.97) 
that  a. person  has  been  hanged  given  that  the  person  is  dead.  Even  though  this  seems  an  unlikely  mistake,  it  is  exactly 
the  kind  of  mistake  that  is  made  with  interpretations  of  statistical  significance  testing  (pp.  384  - 385). 

Evaluation  versus  Research  in  Distance  Education 

While  it  may  have  been  the  intent  of  the  investigators  of  comparison  studies  cited  herein  to  create 
generalizable  findings,  the  motivation  behind  the  studies  were  most  likely  to  obtain  information  about  the  success  of 
local  distance  education  programs.  Appropriate  uses  of  media  comparisons  for  distance  program  evaluation  are 
detailed  as  follows,  as  well  as  alternative  methods  and  exemplary  models. 

Evaluation  in  distance  education 

Program  evaluations  in  education  frequently  look  to  achievement  as  a measure  of  success,  and  sometimes 
through  the  use  of  comparison  studies  as  an  evaluation  method.  Smith  & Glass  (1987)  call  such  inquiries 
comparative  evaluations,  as  the  studies  assess  the  effectiveness  of  a product  or  program  by  pitting  it  against  an 
alternative  product  or  program  that  is  designed  to  meet  the  same  needs.  However,  such  comparisons  work  best  if  the 
treatment  group  and  control  group  are  similar  in  identity  and  can  be  randomly  assigned  (Fitz-Gibbon  & Morris, 
1978),  which  is  usually  not  the  case  in  distance  education.  Participants  in  higher  education  distributed  courses  are 
typically  non-traditional  learners  who  cannot  attend  class  at  the  originating  institution,  hence  their  enrollment  in 
distance  programs.  Not  only  are  these  students  different  demographically,  but  they  also  possess  other  characteristics 
which  vary  from  traditional  college  attendees,  such  as  prior  knowledge  and  experience  and  level  of  motivation 
(Verduin  & Clark,  1991).  In  any  case,  if  comparative  evaluations  can  be  designed  to  represent  comparable  groups  of 
learners,  the  results  of  such  studies  must  be  published  as  local  findings  instead  of  generalizable  contributions  to  the 
theoretical  base  of  distance  education. 

Although  student  achievement  is  one  common  measure  of  distance  program  success,  Keegan  (1996), 
Holmberg  (1989),  and  Thorpe  (1988)  recommend  that  program  evaluators  collect  and  report  a number  of  other  types 
of  data  to  give  the  most  exhaustive  description  of  a distance  education  program.  Saveyne  & Box  (1995)  suggest  the 
collection  of  information  with  regard  to  instructional  design,  participant  attitudes  (student  and  instructor),  and 
implementation  issues,  such  as  technical  quality,  student  support,  etc.  Keegan  (1996)  proposes  a four-point 
evaluation  scheme  for  distance  programs  which  assesses  1)  the  quantity  of  learning  achieved  such  as  the  number  of 
new  students  served,  attrition  rates,  time  to  program  completion,  etc.;  2)  the  quality  of  learning  achieved  measured 
by  the  effectiveness  of  the  program  in  facilitating  desired  learning  outcomes;  3)  the  status  of  the  learning  achieved 
indicated  by  the  transferability  of  program  coursework,  recognition  of  degrees  by  employers  or  graduate  institutions; 
and  4)  the  relative  cost  of  the  learning  achieved  acquired  through  the  analysis  of  the  cost-efficiency  of  distance 
programs  relative  to  conventional  programs,  as  well  as  the  cost  benefits  of  the  distance  program  versus  traditional 
programs  (1996,  pp. 186-188).  The  case  studies  provided  by  Keegan  (1996)  are  mindful  examples  of  distance 
program  evaluations,  as  they  provide  a thorough  portrayal  of  program  efforts  through  analysis  of  the  aforementioned 
indicators.  Another  effective  distance  evaluation  model  can  be  found  in  the  Flashlight  Project  (Ehrmann,  1994),  an 
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effort  by  the  An nen burg/Center  for,Public  Broadcasting  to  heiplnstitutions  of  higher  education  assess  their  uses  of 
instructional  technology, for  distributed  learning.  If  the  intentions  of  investigators  are  to  determine  the  effectiveness 
of  distance  education  programs,  these  evaluation  reports  serve  as  exemplars  due  to  their  comprehensive  approach. 

Research  in  distance  education 

Those  involved  in  the  design,  development,  and  implementation  of  distance  education  programs  have 
access  to  a wealth  of  data  from  which  to  conduct  valid  research.  For  a summary  of  the  existing  literature  base  in 
distance  education,  see  Mclsaac  & Gunawardena  (1996)  and  Schlosser  & Anderson  ( 1994).  Interestingly,  both 
pieces  highlight  the  need  to  move  away  from  the  continued  use  of  media  comparison  studies  toward  more  productive 
lines  of  inquiry.  If  researchers  are  driven  to  investigate  the  effects  of  delivery  media,  perhaps  they  will  heed  Reeves’ 
(1995)  advice  and  design  instructional  technology  studies  that  will  indeed  improve  education.  Examples  of  research 
that  have  served  to  inform  the  development  of  effective  distance  learning  experiences  can  be  seen  in  Garrison  (1990) 
and  Gunawardena,  Campbell  Gibson,  Dean,  Dillon,  & Hessmiller  (1994).  Garrison  (1990)  analyzed  the  ability  of 
audioconferencing  to  provide  necessary  levels  of  interaction  for  feedback  as  well  as  for  student  satisfaction.  Distance 
delivery  media  also  afford  varied  levels  of  social  presence,  as  found  by  Gunawardena,  et  al.  (1994).  Knowing  how 
media  convey  information  and  allow  individuals  to  interact  are  important  considerations  in  the  design  of  distance 
programming.  Indeed,  more  researchers  should  leverage  their  involvement  in  distance  education  experiences  to 
contribute  to  the  knowledge  base  of  the  field.  Mclsaac  and  Gunawardena  (1996)  indicate  that  what  is  needed  is  ’’rich 
qualitative  information  or  programmatic  experimental  research  that  would  lead  to  the  testing  of  research  hypotheses” 
(p.  421).  While  determining  the  efficacy  of  distance  programs  is  important  to  all  stakeholders,  investigators  must 
ensure  that  such  inquiries  begin  with  valid  questions  and  that  the  intentions  behind  the  study  are  well-defined. 
Concurrently,  it  is  equally  important  that  editors  of  professional  journals  also  distinguish  between  research  and 
evaluation  in  distance  education  and  communicate  that  distinction  to  the  authors  of  their  manuscripts. 
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Introduction 

A “ mental  model’'  is  a knowledge  structure,  composed  of  concepts  and  the  relations  between  them 
(Jonassen,  Beissner  & Yacci,  1993;  Shavelson,  1974).  A mental  model  is  a systemic  type  of  learning  outcome  - it  is  a 
system  of  information,  concepts,  and  relationships  between  them.  When  people  have  a mental  model  of  a system,  be 
it  a judicial  system  or  a washing  machine,  they  know  its  components  and  how  the  components  affect  one  another. 

A mental  model  is  a learning  outcome  distinct  from  declarative  and  procedural  knowledge  (Jonassen  & 
Tessmer,  1996),  going  beyond  semantic  relationships  and  skills  acquisition  to  systemic  understanding.  Building  on 
the  theory  base  that  information  in  memory  is  encoded  in  and  retrieved  from  a structure  that  preserves  meaning, 
theorists  posit  mental  models  as  powerful  engines  for  higher  order  cognitive  processes.  If  learners  have  a mental 
model  of  a device  or  system,  the  model  enables  them  to  solve  problems,  generate  inferences,  and  make  predictions 
about  the  system  that  is  modeled  (Johnson-Laird,  1983;  Wilson  & Rutherford,  1989).  Figure  1 models  some  of  those 
functions.  


Figure  1.  Purpose  of  Mental  Models. 
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Rouse  and  Morris’  work  with  troubleshooting  and  mental  models  would  suggest  another  function  of  mental 
models  learning:  predicting  what  is  wrong  with  a system  (diagnosing). 

There  are  at  least  three  important  implications  of  mental  model  research  that  should  be  considered  by 
instructional  designers.  First,  learners  form  mental  models,  whether  the  designer  takes  that  fact  into  account  or  not, 
and  inaccurate  models  can  impede  learning  (Carroll  & Thomas,  1982;  Rouse  & Morris,  1986;  Norman,  1983). 
Second,  troubleshooting  performance  (a  type  of  problem  solving)  can  be  facilitated  through  construction  of  mental 
models  (Gentner  & Gentner,  1983;  Rouse  & Morris,  1986;  Lesgold,  1986;  Mayer,  1989;  Rodgers,  Rutherford,  & 
Bibby,  1992;  Downey,  1996).  Structured  knowledge  has  been  found  to  be  a powerful  predictor  of  the  ability  to 
apply  content  knowledge  to  solve  problems  (Gomez,  Hadfield,  & Housner,  1996;  Cormier  & Hagman,  1987). 

Finally,  research  on  expert  knowledge  representation  indicates  that  structural  knowledge  is  essential  to 
expert  performance  (Chi  & Glaser,  1984;  Larkin,  McDermott,  Simon,  & Simon,  1980).  Tardieu,  Erlich,  & 

Gyselinck,  (1992),  for  example,  found  no  difference  between  expert  and  novices  at  the  propositional  level  of 
memory,  but  significant  differences  at  the  mental  model  level.  That  is,  there  was  little  difference  in  the  amount  of 
domain  knowledge  that  each  group  possessed,  but  there  were  differences  in  the  way  that  knowledge  was  organized. 
Although  the  expert  must  have  a sufficient  knowledge  base  to  draw  upon,  if  it  is  not  expertly  structured,  it  is  of  little 
advantage. 

Mental  Models  Instructional  Strategies 

For  years,  Richard  Mayer  (1984,1997)  has  explored  methods  for  teaching  students  a mental  model  of  a 
simple  system  such  as  a brake  system,  pump,  or  weather  system.  In  general,  his  research  has  indicated  that  graphic- 
textual  representations  of  mental  models  are  more  effective  than  textual  descriptions  (Mayer  & Sims,  1994),  and  that 
animated  graphical  representations  are  more  effective  than  static  ones  (Mayer,  1997).  His  measures  of  mental  model 
performance  are  transfer  of  learning,  inference  and  problem  solving. 

Mayer’s  instructional  methods,  as  well  as  that  of  other  researchers  such  as  Kieras  and  Gentner,  focus  upon 
learners  as  observers  of  a model,  not  as  constructors  of  it.  At  most,  learners  may  manipulate  variables  in  the  model  to 
observe  the  effects  of  their  interaction  (e.g.,  increasing  the  resistance  in  a circuit  model  to  observe  the  effects  on  the 
system). 
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Consequent  with  the  learning  theories  of  constnictivism  (Savery  & Duffy,  1996)  and  constructionism 
(Kafai,  1997),  we  believe  that  mental  models  may  best  be  learned  when  the  learner  has  an  opportunity  to  build  the 
model,  not  just  observe  it.  Research  into  computer-based  learning  indicates  that  learners  understand  and  retain 
information  better  when  they  interact  with  it  in  ways  that  encourage  elaboration,  inference,  or  other  forms  of 
meaningful  learning  (Hannafin  & Peck,  1987).  In  other  words,  building  a model  may  be  more  effective  than  studying 
a model,  whether  static  or  animated. 

The  theoretical  assumption  behind  this  model  building  strategy  is  that: 

Someone  who  acquires  a mental  model  of  a system  acquires  an  understanding  of  the  components  of  a system  and 
their  relation  to  one  another.  This  includes  an  understanding  of  the  function  of  the  component  and  its  effect  on 
ot^er  components. 

From  a constructivist  and  constructionist  perspective,  a learner  can  best  acquire  this  mental  model  by  personally 
constructing  the  model  - by  connecting  the  concrete  components  and  their  abstract  relations 

The  model  construction  task  must  be  scaffolded  by  access  to  cues  and  information,  but  must  not  show  the  model 
itself.  The  most  effective  learning  strategy  involves  students  inferring  the  place  and  function  of  system 
components.  This  building  task  forces  students  to  reflect  on  the  function  of  the  component  and  its  effect 
(relationship)  on  neighboring  components. 

Mental  model  learning  acquisition  is  best  measured  by  three  related  measures:  troubleshooting  diagnosis, 
troubleshooting  solution,  and  prediction  of  change  of  state.  All  three  measures  can  be  used  for  abstract  systems 
(e.g.  judicial  system)  as  well  as  concrete  ones  (pump,  carburetor). 

Prototype  of  the  Instructional  Strategy 

To  explore  the  thesis  that  mental  model  construction  can  be  an  effective  instructional  strategy  for  mental 
models  learning,  we  constructed  a prototype  that  embodies  this  approach.  The  mental  model  to  be  learned  is  the 
Internet,  meaning  the  student  should  learn  the  components  of  the  Internet,  their  functions,  and  their  relationships  to 
one  another.  To  do  this  students  must  learn  to  place  the  components  of  the  Internet  in  their  proper  places,  and 
assemble  the  proper  functions  of  each  component.  For  example,  students  should  learn  the  function  of  a modem,  how 
it  depends  on  its  preceding  component  (the  computer)  for  a signal,  and  how  it  transmits  the  modem  signal  to  the  next 
component  (the  telephone).  This  model-assembly  module  was  constructed  as  a CBT  module,  using  Authorware 
authoring  software.  Learners  must  build  the  model  from  its  disassembled  parts,  and  must  assemble  the  functions  of 
the  parts  as  well  (Figure  1).  The  module  contains  instructions  on  how  to  use  the  system,  the  model  itself,  and  test  for 
acquisition  of  the  mental  model.  We  then  tested  the  module  with  a group  of  learners. 

Prototype  Test 
Subjects 

A group  of  43  subjects  participated  in  a pilot  test  in  November  1999.  All  subjects  were  students  in  the  first 
author’s  undergraduate  Introduction  to  Computers  evening  course  in  a community  college  in  southeastern  Texas. 

The  pilot  test  was  held  during  week  13  of  a 17-week  course. 

Method 

All  students  were  informed  at  the  beginning  of  class  that  the  instruction  for  that  session  would  be  delivered 
by  a computer  animation.  They  were  also  advised  that  the  instructor  wished  to  collect  anonymous  data  on  their 
interaction  with  the  computer  animation  as  a pilot  test  for  his  dissertation  project.  They  were  further  advised  that 
their  participation  in  the  data  collection  was  strictly  voluntary,  and  that  if  they  did  not  wish  to  participate  in  data 
collection  they  could  exercise  that  option  by  not  inserting  the  data  disk  into  their  computer  prior  to  starting  the 
instruction.  All  students  elected  to  participate  in  the  anonymous  data  collection. 

Materials 

Materials  for  the  pilot  test  consisted  of 

1 . the  model-assembly  module,  which  had  been  copied  to  the  hard  drive  of  each  computer  prior  to  the  start  of  class; 

2.  a diskette  containing  a folder  for  the  model -assembly  module  to  write  data  into; 

3.  a brief  paper-and-pencil  background  survey  (age,  sex,  do  you  use  a computer  at  home  or  work,  do  you  use  the  Internet  at 
home  or  work) ; and 

3.  a brief  quiz  with  items  requiring  the  student  to  infer  causes  of  malfunctions,  identify  system  components,  and  infer  system 
functions. 

Procedure 

Students  were  instructed  to  double-click  the  icon  for  the  model-assembly  module  to  open  it,  and  to  go  no 
further  until  the  instructor  had  confirmed  that  all  had  been  opened  successfully.  At  this  point  a random  number 
generator  in  the  Authorware  module  had  assigned  each  subject  to  either  the  control  group  or  the  treatment  group; 
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thus,  assignment  to  groups  was  double-blind.  When  the  instructor  confirmed  that  all  modules  had  opened  properly, 
the  students  were  told  to  proceed  in  accordance  with  the  instructions  they  saw  on  their  screens. 

Both  the  control  group  and  the  treatment  group  went  through  two  computer-based  activities:  an  instructional 
activity  and  a reassembly  activity.  In  the  instructional  activity  the  control  group  observed  the  model  assemble  itself 
by  clicking  on  a Continue  button  as  each  component  of  the  model  was  explained.  Clicking  caused  the  component  to 
go  to  its  assigned  place  and  the  next  element  of  instruction  to  appear.  The  treatment  group  saw  identical  components 
and  instruction,  but  were  required  to  drag  each  component  to  its  correct  location;  if  they  dragged  to  the  wrong 
location,  the  component  snapped  back  to  its  start  position,  and  the  target  spot  was  highlighted  to  facilitate  a second 
try. 

In  the  reassembly  activity  both  groups  saw  identical  screens.  The  components  were  scattered  around  the 
edge  of  the  screen,  and  the  students  were  instructed  to  drag  each  component  to  its  correct  location;  the  locations  were 
consecutively  numbered  to  indicate  both  location  and  sequence.  As  each  component  was  “dropped,”  the  software 
wrote  to  the  diskette  file  the  component  name,  the  time,  and  whether  the  attempt  was  right  or  wrong.  When  all  19 
components  had  been  correctly  placed,  the  module  moved  automatically  to  the  final  screen,  which  advised  the 
student  to  click  on  File,  Quit  to  end  the  program  and  await  further  instruction.  Data  written  to  the  diskette  file  were: 
elapsed  time  in  the  reassembly  event,  number  of  correct  tries,  number  of  incorrect  tries,  and  the  date  and  time. 

Students  were  then  given  the  pa  per- and -pencil  instruments  and  asked  to  read  the  file  number  from  their 
diskettes  (the  random  number  generated  at  the  outset),  write  that  number  on  the  first  page  of  the  questions,  and 
answer  the  questions. 


Results 

The  reassembly  task  was  a recall  test,  requiring  the  student  to  remember  the  basic  location  of  the 
components.  The  quiz  consisted  of  7 items  requiring  the  student  to  infer  causes  of  malfunctions,  identify  system 
components,  and  infer  system  functions.  The  scores  on  the  paper-and-pencil  instrument  and  the  number  of  wrong 
placements  in  the  reassembly  activity  were  compared  using  a pooled  t-test  and  an  ANOVA  in  Data  Desk.  Both  tests 
produced  identical  results: 

Control  Treatment 

Drag  and  drop  errors  n 25  18 

p=0.79  M 23.68  22.17 

SD  17.42  14.87 


Quiz  correct  n 23  16 

p=0.59  M 5.0  5.25 

SD  1.45  1.34 

The  most  significant  data  from  this  pilot  study  was  the  instructor’s  observation  of  several  students  during 
the  reassembly  activity  repeatedly  dragging  a component  to  the  same  wrong  spot.  The  students  apparently  did  not 
understand  that  the  failure  of  the  component  to  snap  into  place  indicated  it  was  being  dragged  to  the  wrong  location. 
The  expressions  on  their  faces  clearly  showed  frustration  (as  opposed  to  indifference  or  vagabonding).  Response  to 
this  phenomenon  is  discussed  below  under  Conclusion  and  Future  Plans. 

The  students  self  reported  the  following  in  the  Background  Survey: 


Age  <18:  1,  18-25:  17,  26-45:  15,  >45:  6 
Sex  f:  17,  m:  23 

Use  computer  at  home  - yes:  34,  no:  5 
Use  Internet  at  home  or  work  - yes:  25  no:  14 

(n  from  the  paper-and-pencil  instruments  is  lower  than  43  because  of  failure  to  complete  portions  of  the  instrument 
and  1 case  of  a student  misreporting  the  random  number  from  the  diskette.) 


The  chart  below  illustrates  an  interesting  result  in  scores  on  errors  in  placement  of  components.  The  over  45 
age  segment  of  the  control  group  had  clearly  higher  error  totals  than  any  other  group.  This  part  of  the  results 
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Figure  2.  Summary  of  Drag  and  Drop  Errors 
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was  not  tested  for  statistical  significance;  however,  in  the  Background  Survey  5 of  6 responded  yes  on  using 
a computer  at  home  or  office,  and  3 of  6 responded  yes  on  using  the  Internet  at  home  or  office. 

Conclusions  and  Future  Plans 

The  data  suggest  some  better  grasp  of  the  model  within  the  treatment  group,  although  the  observed  errors  in 
the  drag-and-drop  tasks  likely  account  for  several  high  error  scores  in  both  groups  in  the  reassembly  task.  This 
failure  to  understand  the  nature  of  the  drag-and-drop  tasks  must  be  addressed  as  the  first  priority.  To  this  end  an 
additional  activity  has  been  placed  between  the  instruction  activity  and  the  reassembly  activity  to  demonstrate  how  to 
drag-and-drop  and  to  emphasize  that  the  movement  of  a component  back  to  its  start  position  means  that  it  was 
dropped  at  the  wrong  location. 

The  data  show  apparently  significant  differences  between  age  groups  and  between  males  and  females  within 
age  groups.  These  results  suggest  the  desirability  of  using  age  and  sex  as  covariates  in  future  studies. 

This  pilot  study  did  not  attempt  to  assess  structural  knowledge.  Future  studies  will  include  structural 
knowledge  comparisons  between  novices  and  experts  using  Pathfinder  techniques. 

The  Background  Survey,  Comprehension  Test,  and  Pathfinder  data  collections  will  all  be  put  into  the 
Authorware  module  so  that  data  collection  will  be  further  streamlined  and  the  data  itself  will  be  entirely  contained  on 
a single  diskette.  Having  all  the  data  written  directly  to  disk  as  it  is  generated  will  eliminate  logistical  distractions 
from  the  experiment  and  will  facilitate  conducting  the  experiment. 

The  participants  in  the  pilot  study  generally  indicated  satisfaction  with  the  instructional  material  in  post- 
experiment, informal  conversations.  They  were  intrigued  by  the  Authorware  product  and  were  eager  to  learn  more 
about  the  Internet.  Their  attitude  suggests  a positive  level  of  receptivity  to  both  the  computer-based  method  and  the 
reassembly  activity  within  it.  This  augurs  well  for  further  studies. 
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Introduction 

This  work  came  out  of  the  practice  of  day  to  day  consultative  instructional  design.  Dealing  with  uncertainty 
in  communicating  with  clients  and  within  the  design  team  can  create  management  headaches,  and  lead  to 
unsuccessful  projects.  Stone  and  Villachica  (1997),  discuss  the  increasing  organizational  risks  of  instructional  design 
efforts,  as  the  size  and  scope  of  instructional  design  (ID)  efforts  increase.  Instructional  designers  are  also  being  asked 
to  develop  projects  that  are  increasingly  complex,  and  developed  in  increasingly  short  timeframes  (Stone  & 
Villachica,  1997). 

Mixed  into  this  difficult  environment  is  the  increasingly  specialized  language  of  both  the  consultant  and  the 
client,  and  technological  complexity.  These  factors  combine  to  create  an  environment  where  miscommunication 
between  client  and  consultant  is  more  likely  and  less  tolerated. 

Fourth  generation  instructional  systems  design  (ISD)  models  advocate  an  iterative  prototyping  regime  in  the 
development  of  instructional  design  projects.  Scenarios  can  be  used  as  one  level  of  prototype,  and  can  create 
possibilities  for  learning  and  increased  communication.  Uncertainty  in  instructional  design  efforts  is  a given. 
Scenarios  are  a tool  to  pin  the  comers  of  uncertainty  that  allow  designers  to  find  a robust  solution. 

What  are  Scenarios? 

In  a strategic  planning  context,  scenarios  are  narratives  of  structurally  different  futures  to  help  guide 
strategic  conversation  and  organizational  learning  (Van  Der  Heijgen,  1996).  In  the  context  of  instructional  design, 
however,  the  use  of  scenarios  is  on  a tactical  level,  not  a strategic  one.  Strategic  conversations  and  the  wider  scope  of 
discussion  should  take  place  before  and  not  during  the  instructional  design  process  begins. 

When  used  on  a tactical  level,  scenarios  are  narratives  that  describe  structurally  and  functionally  different 
approaches  to  solving  a given  problem.  There  are  several  important  points  in  this  definition. 

Scenarios  are  narrative  to  make  the  abstract  concrete.  “End  users  seldom  know  what  they  want  until  they 
see  it”  (Stone  and  Villachica,  1997.  P.  6).  Most  users  are  not  instructional  design  experts,  if  they  were  they  wouldn’t 
need  to  hire  us.  They  know  they  have  a problem,  and  believe  that  an  expert  in  instructional  design  can  help  them 
solve  it.  By  providing  clients  with  stories  of  how  the  intervention  will  be  used  within  their  organization,  IDs  can  help 
all  parties  define  the  scope  and  nature  of  the  problem  and  proposed  solution. 

The  narratives  that  compose  the  different  scenarios,  however,  must  be  structurally  and  functionally  different 
from  each  other  to  present  a range  of  options  to  the  client.  Because  clients  don’t  know  what  they  want  until  they  see 
it,  designers  must  present  a range  of  possibilities  for  the  client  to  see.  Structural  differences  stem  from  differences  of 
theory  or  approach.  Is  the  intervention  constructivist  or  traditional?  Is  it  performance  oriented  or  learning  oriented? 
Functional  differences  stem  from  both  media  and  organizational  contexts.  Is  it  paper  based,  or  computer  based?  Who 
uses  it?  How  often? 


Instructional  Models  and  Scenarios 

The  use  of  scenarios  in  strategic  planning  represents  an  important  shift  from  a rational  approach  to  one  that 
is  process  based  (Van  Der  Heijden,  1996).  The  rational  paradigm  of  strategic  planning  is  a linear  model  that  results 
in  a probabilistic  point  forecast.  Process  models  are  evolutionary  and  iterative.  Taking  their  cue  from  biology,  rather 
than  physics,  process  models  adapt  the  solution  to  fill  an  organizational  niche. 

The  rational  model  of  strategic  planning  has  striking  parallels  to  earlier  models  of  instructional  design.  The 
rationalist  school  assumes  predictability,  clear  intentions,  separation  of  thought  and  action,  full  understanding  and 
rationality  of  human  behavior  (Mintzberg,  1990).  Early  models  of  instructional  design  assume  predictability  of  the 
learner  and  the  design  process,  clear  learning  goals  stated  in  advance,  analysis  before  design  and  development,  clear 
communication  between  client  and  designers,  and  a certain  rationality  in  the  learning  process. 

Experience  with  increasingly  complex  projects,  however,  shows  where  these  assumptions  have  failed, 
creating  a crisis  of  design.  People  are  rarely  predictable,  the  goals  of  a project  are  almost  never  clearly  stated  in 
advance,  analysis  bleeds  into  design  and  development,  and  communication  between  designers  and  clients  is  rife  with 
uncertainty. 

Fourth  generation  models  attempt  to  address  these  shortcomings  (Tennyson  & Foshay,  1995).  They  utilize 
iteration  and  overlapping  design  phases  to  great  effect,  creating  efficiencies  within  the  design  process.  They  model 
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an  evolutionary  learning  process  that  adapts  the  intervention  as  new  information  becomes  available.  Fourth 
generation  process  models  assume  a changing,  uncertain  solution  space.  Scenarios  are  a tool  for  rapidly  prototyping 
multiple  solution  plans  in  an  ISD  effort  in  an  environment  of  uncertainty  and  change. 

Using  Scenarios  in  the  ID  process 

Scenarios  fit  in  any  iterative  design* process  when  there  is  uncertainty,  either  on  the  design  side  or  the  client 
side.  Pinning  the  comers  of  uncertainty  create  a potential  solution  space.  Within  that  space  may  lie  several  near 
optimal  solutions.  The  challenge  of  design  is  to  find  the  highest  possible  peak. 

The  scenario  development  process  can  basically  be  broken  into  three  major  steps: 

1.  Define  the  problem 

2.  Create  a scenario  matrix 

4.  Write  scenario  narratives 

An  example 

A medium  sized  school  district  has  contracted  with  your  instructional  design  group  to  help  them  build  a 
district  wide  information  system.  The  district’s  technology  plan  calls  for  an  integrated  information  system  that  will 
facilitate  communication  between  teachers,  administrators  and  parents.  Beyond  that,  the  district  also  needs  to 
integrate  standards  based  education  using  the  state  frameworks. 

The  superintendent  and  the  district  technology  coordinator  both  know  they  want  something  different  from 
the  packaged  solutions  offered  by  the  vendors.  Most  of  the  solutions  are  inflexible,  and  don’t  integrate  all  of  the 
district’s  requirements.  The  trouble  is,  they  don’t  know  exactly  what  they  want. 

After  meeting  with  the  district’s  team,  you  and  your  group  meet  in  the  conference  room  in  your  office. 
There  are  many  possible  solutions  to  the  problem.  The  district  is  rapidly  expanding  its  physical  network 
infrastructure,  so  there  is  a lot  of  potential. 

Your  team  looks  at  the  possibilities  and  decides  there  are  several  uncertainties  that  will  effect  the  design  of 
the  system.  The  first  question  they  have  is  whether  the  system  is  intended  to  be  used  primarily  by  students,  or  by 
teachers.  The  second  question  the  design  team  has  relates  to  the  district’s  plans  for  innovative  pedagogy.  Expected 
change  in  pedagogy  will  greatly  impact  the  scope  of  the  system. 

These  two  questions  are  the  critical  uncertainties  in  the  design.  There  are  many  other  questions  to  be  asked 
and  answered,  but  the  design  team  feels  that  these  two  questions  are  the  ones  that  will  drive  the  system  in  widely 
divergent  directions  depending  on  the  answers.  The  other  questions  will  derive  from  these  answers.  The  team 
decided  to  create  a set  of  scenarios  around  these  uncertainties,  creating  the  following  axis: 

Figure  1:  Scenario  Matrix 


Teacher  Oriented 


Traditional  Pedagogy 

Innovative  Pedagogy 

Student  Oriented 
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Next  the  team  works  to  fill  in  the  axes  of  the  matrix.  What  are  the  key  differences  between  systems  that  would  exist 
in  each  of  the  four  quadrants?  After  some  discussion,  a consensus  about  the  nature  of  the  different  systems  begins  to 


emerge.  The  matrix  is  filled  in  even  further. 

Figure  2:  Populated  Matrix 

Teacher  Oriented 

o Similar  to  many  products  on  the  market. 

• Mainly  a grade  and  test  tracking  system  for 
administration. 

• Content  broken  out  by  specialty  and  closely  tied  to 
existing  curriculum. 

• Capture  current  best  practices. 

• Performance  support  orientation  for  teachers. 

© Create  systems  for  knowledge  management. 

• Content  in  flexible  database,  easily  searched  and 
rearranged. 

Traditional  Pedagogy 

Innovative  Pedagogy 

• Common  CBT  and  IMS  system. 

• Content  arranged  in  order,  presented  in  order. 

• Automated  grading  and  tracking  functions. 

• Data  collection  and  analysis  at  higher  levels  of 
organization. 

• Flexible  content  organization,  easy  for  students  to 
explore. 

• Student  research  and  exploration  tools. 

• Opportunities  for  student  creativity  and  sharing. 

Student  Oriented 


Once  all  of  the  quadrants  are  filled  in,  it’s  time  to  start  writing  the  narrative.  The  team  decides  to  delegate 
the  narrative  for  the  upper  right  quadrant  to  you.  You  decide  to  write  it  from  the  perspective  of  a new  teacher,  just 
starting  to  use  the  system. 

Augmented  Teaching 

Jeanne  breathed  a sigh  of  relief.  Her  morning  was  over  and  now  she  had  a minute  to  gather  her  thoughts. 

She  was  trying  to  figure  out  how  to  support  a few  students  in  her  class  that  she  realized  didn’t  have  the  prerequisite 
knowledge  to  do  the  work  in  her  9th  grade  English  class.  She  was  also  concerned  about  trying  to  tie  her  curriculum 
into  the  pod’s  science  and  technology  focus.  There  were  four  pods  in  the  high  school,  each  acting  as  it’s  own  magnet 
school.  She  had  recently  joined  the  district  and  was  trying  to  relate  her  English  curriculum  to  the  larger  science 
focus. 

She  turned  to  her  screen  and  opened  the  Teachers  Assistant.  The  program,  or  T A as  it  was  affectionately 
referred  to,  was  a combination  communications  system  and  resource  base  for  all  of  the  district’s  teachers.  It  also 
linked  to  other  school  districts  so  teachers  from  all  over  the  state  could  share  experience  and  resources.  It  also  held 
her  grade  book;  the  results  of  all  of  her  students  last  state  standards  exam,  as  well  as  IEPs,  parent  contact  information 
and  a wealth  of  other  district  information. 

After  logging  on,  she  opened  up  the  query  system  and  typed  in  “teaching  English  and  science’’.  The  results 
showed  an  ongoing  statewide  conference  on  that  subject,  several  lesson  plans  and  teaching  tools  tied  to  state 
standards.  She  clicked  on  one  of  the  lesson  plans  that  looked  interesting. 

The  lesson  plan  gave  recommendations  and  a scoring  guide  for  a writing  exercise  on  the  history  of  science. 
Students  could  choose  a specific  idea  or  scientist  and  write  about  that  subject.  She  liked  the  idea  and  the  scoring 
guide  helped  her  tie  it  to  the  state  standards.  Clicking  on  the  communication  toolbar,  she  quickly  composed  an  email 
to  both  the  team  history  teacher  and  science  teachers  proposing  they  discuss  the  idea  further  as  a team  project. 

The  history  teacher  immediately  emailed  her  back.  He  liked  the  idea  and  had  been  thinking  about  something 
similar.  In  fact,  he  was  planning  to  base  the  entire  semester  around  the  history  of  ideas.  Jeanne  immediately  thought 
about  the  TV  series  “Connections”.  It  would  be  a great  way  to  introduce  the  idea  to  the  kids.  She  clicked. on  the 
district  resources  window  and  searched  the  district’s  video  library.  The  library  had  a few  episodes  on  tape!  She 
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mailed  the  history  teacher  back  with  the  reference  number  for  the  videotapes  and  made  an  appointment  to  meet  him 
in  the  break  room  at  lunch. 

After  you  have  written  the  scenario,  reread  it  to  make  sure  you  have  hit  the  major  points  in  your  outline.  Has 
each  been  properly  addressed?  Can  you  picture  what  each  of  those  abstract  points  means  in  terms  of  concrete 
interventions? 

After  the  scenarios  are  written  and  named,  the  team  presents  them  to  the  client.  Being  a skilled  practitioner, 
you  are  deliberately  present  the  scenarios  as  a range  of  options,  not  as  given  solutions.  You  are  especially  careful  not 
to  present  one  scenario  as  favored  over  the  others. 

The  clients  and  the  design  team  discuss  each  scenario  and  it's  implications.  The  team's  listens  to  the  clients 
to  discern  the  points  of  each  scenario  that  meet  their  needs,  and  the  parts  that  don't.  From  this  discussion,  the  design 
team  and  the  clients  together  generate  the  next  list  of  requirements. 

The  scenario  generation  process 

1 . Define  the  question:  What  is  it  you  are  trying  to  answer  with  the  conversations  generated  by  the  scenarios. 

2.  Identify  critical  uncertainties:  These  may  be  obvious  in  the  course  of  discussion,  or  they  may  emerge  out  of  structured 
group  processes  like  brainstorming. 

3.  Create  scenario  matrix:  By  combining  the  uncertainties,  a matrix  of  possible  outcomes  is  created. 

4.  Characterize  the  quadrants:  Create  a few  bullet  points  of  the  defining  characteristics  of  the  quadrant.  Remember  that  the 
scenarios  that  emerge  from  each  quadrant  should  be  structurally  or  functionally  different  to  present  a range  of  options.  If 
all  of  the  scenarios  look  the  same,  you  have  created  a point  solution  that  may  not  be  optimal. 

5.  Create  the  scenario  narratives:  Using  the  outlines  from  step  4,  create  a narrative  of  the  use  of  the  system  within  the 
organization.  Be  sure  to  be  as  specific  as  possible.  It  is  effective  to  write  the  narrative  from  the  point  of  view  of  the  user. 

Be  sure  to  make  all  of  your  assumptions  explicit  in  the  narrative.  Who  is  the  user?  What  are  their  goals?  What  resources  do 
they  have?  What  do  they  see  when  using  the  system? 

6.  Name  the  scenarios:  Give  each  scenario  a unique  and  memorable  name.  This  will  help  the  clients  remember  and  discuss 
them.  It’s  easier  to  refer  to  the  “Flock  of  seagulls”  scenario  than  it  is  to  remember  what  "Scenario  1 ” involved. 

7.  Present  the  scenario  options  to  the  client:  When  presenting  the  scenarios  to  a client,  remember  that  the  matrix  represents  a 
solution  space.  The  client’s  most  robust  solution  probably  lies  somewhere  between  the  boundaries  set  by  the  scenarios. 
Having  a preferred  scenario  can  limit  the  conversation. 
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The  Major  Themes  and  Trends  in  Doctoral  Dissertation  Research  in  Educational 

Technology  from  1977  through  1998 

Edward  P.  Caffarella 
University  of  Northern  Colorado 

Each  year  an  average  of  about  120  doctoral  students  graduate  from  55  educational  technology  programs 
at  universities  across  the  United  States.  Each  of  these  students  completes  a dissertation  as  part  of  their 
initial  preparation  as  a researcher  in  the  field.  The  topics  for  these  dissertations  vary  widely  with  no 
standard  dissertation  type , topic,  nor  methodology  within  the  field.  This  variability  is  reflection  of  the 
educational  technology  field  and  indicative  of  the  multiplicity  of  research  concerns  for  the  students  and 
their  faculty  advisors . 

Statement  of  Purpose 

The  purpose  of  this  research  project  was  to  review  the  dissertations  that  have  been  done  in  the  field  from 
1977  through  1998.  During  this  period  of  time  more  than  2,689  dissertations  have  been  completed  at  55  United 
States  institutions.  Approximately  87%  of  these  dissertations  were  completed  at  the  25  larger  programs.  This  review 
reveals  what  has  been  done  in  doctoral  programs  and  student  interest  over  the  past  two  decades.  The  specific 
research  questions  addressed  were: 

What  are  the  major  themes  over  the  past  22  years? 

1 . How  have  the  themes  changed  over  the  22  years? 

2.  How  have  the  research  methodologies  changed  over  the  past  22  years? 

3.  What  trends  are  evident  in  the  field? 

4.  What  are  the  emerging  trends  in  dissertation  research? 

The  answers  to  these  questions  lead  to  a better  understanding  of  doctoral  dissertation  research  in  the  field  of 
educational  technology  but  also  an  understanding  of  the  evolving  nature  of  the  field  as  reflected  in  the  research. 

Methodology 

The  methodology  used  in  this  study  was  a content  analysis.  The  use  of  content  analysis  enabled  the 
researcher  to  identify  major  constructs  within  the  dissertation  research  and  to  classify  these  constructs.  Content 
analysis,  as  described  by  Weber  (1990),  has  been  used  in  a number  of  studies  to  show  trends  within  a field  or  body 
of  literature.  The  basic  premise  of  content  analysis  is  to  classify  words  of  text  into  content  categories.  This  technique 
was  used  by  Ely  (1996)  in  his  series  of  reports  on  trends  in  the  field.  Insch,  Moore,  and  Murphy  (1997)  provide 
further  support  that  content  analysis  is  an  appropriate  methodology  for  analyzing  written  textual  materials.  Evans 
(1996)  describes  recent  advances  in  computer  software  for  supporting  content  analysis. 

The  Doctoral  Research  in  Instructional  Design  and  Technology:  A Directory  of  Dissertations , 1977-1998 
(Caffarella,  1999)  database  of  dissertations  was  used  in  this  study.  This  database  is  available  on  the  Web  at 
http://www.edtech.unco.edu/disswww/dissdir.htm.  The  database  directory  is  divided  into  five  major  sections 
including  the  student  listing,  key  word  in  context  index,  institution  listing,  chairperson  listing,  and  year  listing.  The 
student  section  lists  the  student’s  name,  graduation  year,  dissertation  title,  institution,  and  chairperson.  The  key  word 
in  context  index  makes  it  possible  to  look  up  any  of  the  major  words  in  a title.  The  institution  section  lists  all  of  the 
dissertations  completed  at  a specific  institution.  The  chairperson  section  lists  all  of  the  dissertations  directed  under 
the  guidance  of  a particular  chairperson.  The  year  index  lists  all  dissertations  completed  in  a particular  year.  These 
indexes  provide  access  to  the  dissertation  data  in  a variety  of  ways. 

The  data  for  the  directory  is  supplied  directly  by  the  universities  in  the  United  States  offering  graduate 
programs  in  instructional  design  and  technology.  These  various  programs  are  offered  under  a wide  variety  of  titles 
including  educational  technology,  instructional  development,  educational  media,  instructional  technology,  and 
instructional  systems.  The  basic  list  of  institutions  was  taken  from  the  list  of  doctoral  programs  in  the  Educational 
Media  and  Technology  Yearbook  1996  (Ely  & Minor,  1996).  This  list  was  checked  against  the  membership  list  for 
the  Professors  of  Instructional  Design  and  Technology  for  the  locations  of  additional  programs.  The  intent  was  to  be 
as  inclusive  as  possible  so  that  the  wide  variety  of  dissertations  in  the  field  of  instructional  design  and  technology 
would  be  included  in  the  directory.  The  decision  to  participate  and  which  dissertations  to  submit  was  left  to  the 
individual  institutions. 

The  directory  currently  contains  entries  for  dissertations  completed  by  2,689  students  during  the  period 
from  1977  through  1998.  As  shown  in  figure  1,  the  dissertations  are  distributed  relatively  evenly  over  the  years  with 
an  average  of  approximately  120  dissertations  completed  each  year.  The  numbers  shown  for  1997  and  1998  are 
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probably  underestimates  of  the  actual  number  of  dissertations.  Due  to  the  way  dissertations  are  reported  the 
institutions  tend  to  be  delayed  in  forwarding  the  entries  and  there  are  always  additions  to  the  database  that  are  for 
dissertations  completed  two  to  three  years  earlier. 

Figure  1.  Number  of  Dissertations  Completed  for  Each  Year  from  1977  through  1998 


The  dissertations  were  coded  using  multiple  coding  points  to  identify  the  major  themes  and  constructs 
within  the  field.  The  QSR  NUD*IST  software  was  used  for  the  coding  and  the  analysis.  The  coding  was  an 
evolutionary  process  of  data  extraction  and  reduction  to  identify  both  major  themes  and  trends  within  the  dissertation 
research. 

Ely  (1996)  did  a similar,  but  different,  content  analysis  study  of  literature  sources  in  the  field.  His  study  was 
limited  to  one  year  but  included  journals,  dissertations,  conferences,  and  ERIC  documents  with  the  overall  purpose 
of  identifying  trends  for  a single  year,  1995.  Among  his  documents  were  37  dissertations  from  five  large  doctoral 
programs  completed  during  that  year.  This  study  builds  upon  the  Ely  study  but  looks  at  only  dissertations  for 
multiple  years. 

Findings 

Initial  analysis  of  the  dissertation  data  showed  a wide  variety  of  topics  being  investigated  by  students  in  the 
educational  technology  field.  This  is  indicative  of  the  eclectic  nature  of  the  field  of  educational  technology  and  the 
widely  divergent  topics  that  are  included  in  the  field.  There  are  no  predominate  themes  or  topics  for  the  dissertation 
research.  There  were,  however,  several  themes  that  were  prominent  in  the  data  that  will  be  discussed  in  the  following 
section. 

Clearly  the  most  popular  topic  was  research  on  and  about  computers.  This  took  many  forms  including 
appropriate  uses,  software  design,  individual  differences,  as  well  as  the  effectiveness  of  computers  as  a teaching  tool. 
As  shown  in  Figure  2,  the  number  of  studies  dealing  with  computers  grew  with  the  introduction  of  microcomputers. 
The  Apple  II  was  introduced  during  the  late  70s,  the  IBM  PC  arrived  in  the  early  80s,  and  the  Macintosh  was 
released  in  the  mid  80s.  With  the  introduction  of  each  of  these  technologies  there  was  an  increase  in  the  number  of 
studies  and  for  the  past  14  years  between  15  percent  and  25  percent  of  all  dissertations  have  dealt  with  computers. 
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Figure  2.  Percentage  of  Dissertations  Dealing  with  Computers  for  Each  Year 


One  theme  that  was  constant  throughout  the  22  year  period  was  research  on  instructional  development, 
instructional  design,  and  instructional  systems  development  (see  Figure  3).  This  research  amounted  to  approximately 
two  percent  to  five  percent  of  the  dissertations  in  a given  year.  This  was  not  an  overwhelming  number  of 
dissertations  but  was  the  most  constant  theme  of  interest  over  the  years. 

Figure  3.  Percentage  of  Dissertations  Dealing  with  Instructional  Development, 
Instructional  Design,  or  Instructional  Systems  Development  Each  Year 
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Two  other  themes  seemed  to  persist  throughout  the  period  although  both  had  years  of  popularity  and  years 
with  very  little  research.  These  themes,  shown  in  figure  4,  are  (a)  simulation  and  games  and  (b)  television  and  video. 
The  kinds  of  research  done  on  these  topics  varied  greatly  over  the  22  years.  The  early  simulation  and  game  research 
was  largely  process  and  design  related  while  the  more  recent  research  has  looked  at  simulations  and  games  within  a 
computer  environment.  Likewise,  the  television  research  followed  the  evolution  and  change  of  this  technology. 

Figure  4.  Percentage  of  Dissertations  Dealing  with  Simulation  and  Television  for  Each  Year 


□ Simulation 
■ Television 


As  might  be  expected,  at  any  given  point  in  time  there  is  research  dealing  with  the  newest 
hardware/software  technologies.  For  example,  many  studies  in  the  late  seventies  dealt  with  the  use  of  film  and  video 
but  more  contemporary  studies  have  dealt  with  computer  applications.  The  growth  of  research  on  computing,  shown 
in  figure  1,  is  one  example  of  this  change.  The  changing  nature  of  the  field  of  educational  technology  and  the 
tendency  of  dissertation  research  to  follow  the  new  developments  can  be  seen  clearly  in  Figure  5.  During  the  early 
years  film  research  was  a major  segment  of  dissertation  research.  During  the  middle  years  research  utilizing 
videodiscs  was  popular  amongst  doctoral  students.  In  recent  years  multimedia  and  hypermedia  have  been  major 
topics  for  research. 
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Figure  5.  Percentage  of  Dissertations  Dealing  with  Film,  Videodisc, 
Multimedia,  and  Hypermedia  for  Each  Year 


□ Film 

B Videodisc 

□ Multimedia 

□ Hypermedia 


The  research  methodologies  used  in  the  dissertations  show  some  interesting  trends.  The  most  startling  trend 
is  the  reduction  in  the  number  of  comparison  studies  from  1977  until  1998  as  shown  in  Figure  6.  Many  of  these 
studies  compared  one  medium  against  another  medium  to  determine  if  one  medium  was  inherently  better  than  the 
other.  In  1983  Clark  (1983)  wrote  about  the  problems  with  comparison  studies  and  launched  a debate  that  continued 
into  the  early  90s.  During  the  late  70s,  comparison  studies  represented  around  1%  of  all  dissertation  research.  During 
the  late  80s  and  early  90s  the  number  of  comparison  studies  had  dropped  to  around  5%.  For  the  last  five  years 
comparison  studies  have  accounted  for  less  than  3%  of  dissertation  studies. 


Figure  6.  Percentage  of  Dissertations  Using  Comparative  Research  Methodology 

for  Each  Year 
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Another  clear  shift  in  research  methodology  can  be  seen  in  a reduction  in  the  number  of  experimental 
studies  and  an  increase  in  the  number  of  qualitative  studies.  In  the  late  70s,  very  few  qualitative  studies  were  done  by 
students  (see  Figure  7)  and  these  studies  were  clearly  the  exceptions  to  the  norm.  Now  dissertations  with  qualitative 
research  designs  make  up  a substantial  proportion  of  the  studies  and  exceed  the  number  of  experimental  studies. 

Figure  7.  Percentage  of  Dissertations  Using  Experimental  and  Qualitative 
Research  Methodologies  for  Each  Year 


□ Experimentel 
■ Quelitatrve 


Institutions  Offering  Doctoral  Programs 

There  were  55  institutions  who  graduated  students  with  doctoral  degrees  in  educational  technology  during 
the  period  1977-1999.  The  programs  with  the  most  graduates  included  Indiana  University,  Boston  University,  the 
University  of  Southern  California,  Syracuse  University,  and  Florida  State  University.  Figure  8 shows  the  relative 
numbers  of  graduates  for  the  25  institutions  with  the  largest  numbers  of  graduates.  These  institutions  graduated  2,287 
students  while  the  remaining  30  institutions  graduated  only  402  students.  There  is  a clear  variability  in  the  size  of  the 
doctoral  programs  with  many  extremely  small  doctoral  programs.  Interestingly  only  four  of  the  25  are  private 
institutions  with  the  vast  majority  of  large  educational  technology  programs  offered  at  public  institutions. 

Figure  8.  Twenty-Five  Programs  with  the  Largest  Number  of  Graduates 
from  1977  through  1998 
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Figure  9 shows  the  number  of  graduates  over  the  past  five  years  for  the  25  largest  programs.  The  collection 
of  institutions  is  essentially  the  same  as  the  list  in  Figure  8 but  the  order  of  the  institutions  is  different  with  some 
gaining  in  the  ranking  and  others  moving  to  the  right.  For  the  last  five  years,  these  25  institutions  graduated  483 
students,  which  is  95%  of  the  total  graduates. 

Figure  9.  Twenty-Five  Programs  with  the  Largest  Number  of  Graduates 
for  the  Past  Five  Years 


Chairpersons  of  Dissertations 

The  25  chairpersons  with  the  largest  numbers  of  doctoral  graduates  from  1977  through  1998  are  shown  in 
Figure  10.  The  institutions  for  these  chairs  are  reflected  in  institutions  shown  in  Figure  8.  These  25  chairs  worked 
with  1,018  students  during  this  22  year  period  of  time  which  represents  37%  of  the  total  students.  There  were  205 
chairs  who  chaired  only  one  dissertation  each.  Clearly,  the  advisement  of  large  numbers  of  doctoral  students  is  being 
done  by  a relatively  small  number  of  educational  technology  professors. 

Figure  10.  Twenty-Five  Chairs  with  the  Largest  Number  of  Graduates  from  1977  through  1998 
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The  cohort  of  chairs  with  the  largest  number  of  graduates  over  the  past  five  years  (see  figure  11)  includes  a 
few  new  individuals.  The  27  professors  in  this  group  chaired  271  out  of  a total  of  518  dissertations  representing  52% 
of  the  total.  Many  of  these  professors  have  actively  chaired  dissertations  over  the  entire  22  years  but  there  are  several 
new  names  who  have  assumed  responsibilities  for  supervising  dissertations. 

Figure  1 1 . Twenty-Seven  Chairs  with  the  Largest  Number  of  Graduates 
for  the  Past  Five  Years 
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Summary 

The  field  of  educational  technology  is  a very  varied  field  as  reflected  in  the  research  of  doctoral  students. 
The  students  select  a range  of  topics  that  have  no  particular  theme  not  pattern  with  the  research  following  the 
development  of  new  technologies.  The  field  utilized  a variety  of  research  methodologies  with  qualitative 
methodologies  having  a major  role  during  the  past  few  years.  Most  of  the  students  graduate  from  a relatively  small 
number  of  institutions  although  there  are  55  institutions  offering  doctoral  programs  in  educational  technology. 
Likewise,  a relatively  small  number  of  professors  chair  most  of  the  doctoral  dissertations.  Thus,  although  the  field  is 
broadly  based  and  draws  upon  many  disciplines,  the  doctoral  study  is  highly  concentrated  with  a few  institutions  and 
professors. 
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A Case  Study  of  Using  Groupware  to  Support  Collaborative  Activities  in  a Distance 

Learning  Course 

Lara  M.  Luetkehans,  PhD 
Northern  Illinois  University 


Abstract 

This  case  study  examined  the  ways  a groupware  tool , TWISTER , was  used  to  create  a learner-centered 
distance  learning  environment.  Data  were  collected  using  learner  participant  surveys,  observation  field 
notes,  interviews,  computer  transcripts,  and  course  artifacts.  Data  analysis  focused  on  participant  use  of 
TWISTER,  characteristics  of  the  participants  and  learning  environment  that  influenced  participation , and 
the  communication  and  interaction  in  the  different  components  of  the  learning  environment. 

Findings  indicated  that  participants  recognized  limitations  in  interactivity  frequency  and  levels  in  the 
compressed-video  environment.  Planned  activities  successful  in  this  environment  included  using  a wide 
range  of  activities  and  providing  several  means  of  access  for  communication.  Although  open  and 
collaborative  activities  encouraged  rapport  building  and  enhanced  interaction , participants  still  identified 
challenges.  Use  of  TWISTER  to  assist  in  facilitating  the  student-centered  activities  also  presented 
challenges.  The  tool  did  not  support  the  activities  as  intended.  There  was  low  overall  usage  of  the  tool  by 
all  participants.  For  the  participants,  the  added  value  of  the  features  afforded  by  groupware  did  not 
outweigh  the  convenience  of  email.  Existing  habits  and  limitations  in  functionality  and  interface  were  likely 
contributors  to  this  low  use. 

Results  suggested  that  distance  learning  environments  continue  to  need  to  be  improved  through  devising 
learner-centered  activities  and  increasing  interaction.  Recommendations  for  practice  included:  I) 
assessing  and  preparing  the  learners  for  the  learning  environment,  2)  incorporating  ongoing  community 
building  and  other  short  activities,  3)  planning  extra  time  for  management  issues,  training,  and  practice, 
and  4)  selecting  groupware  tools  based  on  functionality  in  a learning  environment  and  the  types  of 
activities  to  be  supported. 

Introduction 

Post-secondary  institutions  are  rapidly  adopting  distance  modes  of  educational  delivery  because  of 
technological  innovations,  but  more  than  the  technology  needs  to  be  explored.  The  teaching  and  learning  also  needs 
to  be  examined  (Mclsaac  & Gunawardena,  1996).  All  too  often  these  new  technologies  are  used  to  present  one-way 
lectures  to  students  in  remote  locations  (Jonassen,  Davidson,  Collins,  Campbell,  & Haag,  1995).  Contemporary 
perspectives  in  teaching  and  learning  theory  emphasize  student-centered  pedagogies  where  students  take  part  in  the 
development  of  their  own  learning  experiences  (Jonassen,  Davidson,  Collins,  Campbell,  & Haag,  1995;  Savery  & 
Duffy,  1995;  Scardamalia  & Bereiter,  1994;  Spiro,  Feltovich,  Jacobson,  & Coulson,  1991). 

Interactions  among  students  and  instructors  are  a vital  piece  of  student  centered  learning.  Despite 
innovations  in  distance  learning  technologies,  student-centered  models  of  learning  are  difficult  to  implement  in 
distance  learning  environments.  The  communication  among  participants  and  levels  of  interaction  are  still  a concern. 
Problems  lie  not  only  in  how  the  technologies  are  being  used,  but  also  there  remains  some  limitations  in  the 
technologies  themselves.  For  example,  compressed  video  delivery  systems  have  been  touted  as  a premier  interactive 
distance  learning  technology,  however,  studies  show  that  interaction  and  communication  is  still  hampered  making  the 
use  of  collaborative  learning  activities  rare  (Comeaux,  1995;  Hamish,  Reeves,  & Noah,  1996;  Henri  & Rigault,  1991; 
Kolomeychuk  & Peltz,  1992;  McHenry  & Bozik,1995;  Thomerson  & Smith,  1996). 

Today’s  technological  developments  can  enhance  communication  among  the  participants  of  a course. 
Through  computer  mediated  communication  tools  such  as  email  and  conferencing,  it  may  be  possible  to  create 
environments  that  do  not  require  face-to-face  meetings  as  the  only  means  for  achieving  student-centered, 
collaborative  learning  models. 

Groupware  tools  may  be  one  such  technology.  Groupware  “help[s]  groups  structure  work  through  group 
decision  support  systems,  project  management  tools,  electronic  conferencing  systems,  and  shared  editors”  (Jonassen, 
Davidson,  Collins,  Campbell,  & Haag,  1995,  p.  17).  This  research  sought  to  identify  ways  in  which  groupware  tools 
could  be  used  to  support  collaborative  activities  in  a distance  learning  environment. 

Groupware  systems,  in  general,  differ  from  other  Internet  conferencing  tools  like  email/Listservs  and 
newsgroups  in  several  ways.  Email/Listservs  have  little  if  any  structure  to  the  messaging.  There  may  be  several 
topics  being  discussed  simultaneously.  Messages  of  varying  topics  can  be  sent  sequentially.  This  requires  the  user  to 
create  their  own  structure  upon  receipt.  Listservs  only  exacerbate  this  problem  with  many  having  a heavy  message 
load.  Email/Listservs  are  referred  to  as  "push”  technologies  because  the  information  is  sent  to  the  user.  Another  of 
type  is  "pull"  technology.  Newsgroups  are  "pull"  technology.  With  this  form  of  conferencing  the  user  goes  to  an 
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internet  location  where  the  conference  is  maintained.  "Pull"  technology  solves  some  of  the  structuring  and  load 
problems.  Newsgroups  provide  some  threading  of  messages  and  users  can  select  which  messages  they  would  like  to 
read.  (Dennis,  Pootheri,  & Natarajan,  1996). 

Given  the  added  functionality  and  support  for  group  work  that  groupware  provides,  researchers  hoped  that 
student-centered  models  of  learning  might  be  applied  to  post-secondary  learning  environments  where  participants  are 
separated  by  space  and/or  time.  Based  on  a review  of  current  perspectives  in  the  areas  of  distance  learning,  open- 
ended  learning,  and  collaborative  learning,  the  following  features  were  identified  as  important  to  selecting  a 
groupware  product. 

The  WWW  is  a relatively  accessible  technology  to  participants  affiliated  with  a higher  education  institution 
and  was  therefore  considered  exclusively  as  the  delivery  medium.  The  WWW  also  offers  a familiar  environment, 
thereby  presenting  only  a slight  learning  curve  to  most  participants.  Past  groupware  systems  were  often  proprietary 
systems  run  on,  LANS  or  WANS.  Recently,  many  Internet-based  systems  have  been  made  available. 

Other  desired  features  of  the  groupware  included: 

1 . A space  for  presenting  an  anchor  or  macrocontext  can  assist  learners  in  an  environment  that  enables  open  exploration  into 
a set  of  concepts  or  set  of  skills  (CGTV,  1992). 

2.  Participants  should  be  able  to  engage  in  both  synchronous  and  asynchronous  communication.  The  added  capabilities  to 
thread  discussions  and  link  ideas  can  provide  metacognitive  support. 

3.  Databases  of  information  provide  resources  for  accessing  content.  These  could  include  both  pre -developed  resources  as 
well  as  those  produced  through  participant’s  prior  learning  such  as  a case  library. 

4.  By  giving  learners  the  ability  to  add  or  post  group  products  they  contribute  to  the  knowledge  base  of  a larger  group  and 
also  see  that  their  products  have  value.  (Scardamalia  & Bereiter  1994;  Lebow,  Wager,  Marks,  & Gilbert  1996). 

5.  Hypertext  links  will  provide  a means  for  practicing  and  learning  cognitive  flexibility.  It  also  enables  the  idea  linking 
described  above  (Spiro,  Feltovich,  Jacobson,  & Coulson  1991). 

6.  Writing/production  tools  enable  groups  to  produce  evidence  of  their  knowledge  and  expertise. 

7.  Metacognitive  process  structures  can  be  built  into  the  environment  as  a coaching  mechanism  until  learners  are  comfortable 
directing  their  own  learning. 

8.  Shared  workspace  allows  collaboration  among  individuals  as  they  strive  to  construct  meaning  of  the  concepts  or  practice 
skills.  For  example,  this  could  be  in  the  form  of  a synchronous  whiteboard. 

The  web-based  system  used  in  this  study  was  TCBWorks.  This  groupware  product  was  designed  to  support 
writing,  synchronous  and  asynchronous  discussion,  idea  generation,  and  decision-making  (Dennis,  Pootheri,  & 
Natarajan,  1997).  Because  it  did  not  provide  all  of  the  desired  features,  TCBWorks  was  incorporated  into  a class 
web  site  that  added  some  functionality.  The  web  site,  TWISTER,  Talking  - Writing  - Information  access  - Solving 
problems  with  - Technology  for  - Education  and  Research  (no  longer  available  online.)  integrated  a variety  of  tools 
including  TCBWorks,  a groupware  product,  as  well  as  simultaneous  Internet  access  to  information  resources,  course 
information,  and  other  related  resources.  See  Figure  1. 

Figure  1 TWISTER  Tools 


Methodology 

The  aim  of  this  study  was  to  create  and  examine  a student-centered  distance  learning  environment 
facilitated  by  using  a groupware  tool  as  a supplement.  Naturalistic  data  collection  and  analysis  were  used  in  order  to 
gain  an  understanding  of  the  participants’  experience  and  to  identify  considerations  for  using  a groupware  tool  to 
support  a distance  learning  course.  Given  the  nature  of  qualitative  research,  the  questions  with  which  I began  this 
study  were  open  enough  to  allow  for  a flexible  research  design  and  the  possibility  of  unexpected  outcomes.  As  the 
study  progressed,  the  following  questions  guided  my  data  analysis. 
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Question  1 . What  factors  (i.e.  participants'  personal  characteristics,  attitudes,  skills,  or  circumstances)  influence  participation  within 
the  three  learning  environments  (in-class,  COMPRESSED  VIDEO,  and  TWISTER)? 

Question  2:  In  what  ways  do  the  participants  (instructor,  teaching  assistant,  and  students)  make  use  of  the  GROUPWARE  tool? 


Question  3.  What  is  the  communication  and  interaction  among  the  participants  like?  Are  there  patterns  of  interaction  that  emerge  in 
the  different  learning  environments? 


This  study  looked  at  the  complex  phenomena  that  occur  in  a distance  learning  course  supplemented  with 
computer  supported  collaborative  learning  activities.  The  participants’  behavior  in  such  a complex,  dynamic 
situation  is  not  as  easily  controlled  as  in  a laboratory  style  study.  In  order  to  study  the  uniqueness  and  complexity  of 
this  situation,  the  whole  phenomena  was  investigated  in  an  authentic  environment  (Jordan,  1996;  Lecompte  & 
Preissle,  1993;  Patton,  1990).  Through  qualitative  case  study,  a deep  understanding  of  the  context,  the  participants, 
and  the  interactions  among  them  emerged. 

This  research  study  examined  a telecommunications  in  education  course  offered  spring  1997.  This  course 
met  for  a ten-week  term  at  two  different  sites  simultaneously  with  each  classroom  connected  via  compressed  video 
technology.  In  addition,  other  class  activities  took  place  both  synchronously  and  asynchronously  through  a 
groupware  tool,  TWISTER.  This  format  falls  under  what  rlarasim,  Hiltz,  Teles,  and  Turoff  (1995)  call  mixed  mode, 
in  which  the  class  meetings  are  extended  and  supported  by  computer  mediated  activities  and  discussions. 

During  the  first  two  weeks,  data  were  collected  to  provide  a baseline  profile  of  the  students,  instructor,  and 
class  environment  through  student  pre-surveys,  an  interview  with  the  instructor,  and  in-class  observations. 

Throughout  the  quarter  the  instructor,  teaching  assistant  and  researcher  held  weekly  debriefing  sessions.  During  each 
class  session  the  research  captured  written  field  notes  from  alternating  sites.  In  the  last  two  weeks  of  the  course, 
student  participants  were  interviewed  individually  and  in  their  collaborative  work  groups.  Finally,  a post-survey  was 
given  to  the  students.  Both  the  instructor  and  teaching  assistant  were  interviewed  at  the  conclusion  of  the  course.  All 
of  the  instruments  were  developed  specifically  for  the  study.  The  tasks  used  in  the  class  projects  were  co-developed 
by  the  researcher,  the  instructor,  and  teaching  assistant. 

There  was  a wide  range  of  activities  planned  for  this  class.  There  were  individual  and  collaborative, 
technology  supported  and  technology  required,  in-class,  and  independent.  These  activities  also  included  a variety  of 
different  kinds  of  tasks  including  idea  generation,  problem  solving,  construction/production,  and  hands-on.  These 
varying  activities  were  built-in  to  meet  the  needs  of  different  learners  with  varying  skill  levels.  Also,  for  research 
purposes,  the  wide  variety  offered  an  opportunity  to  examine  a breadth  of  activities  both  supported  and  not  supported 
with  compressed  video  and  TWISTER. 

During  a class  session  it  was  typical  for  participants  to  engage  in  a discussion  of  the  readings  led  by  fellow 
students,  one  from  each  site.  This  might  be  followed  by  a guest  lecturer  or  a presentation  from  one  member  of  the 
instructional  team.  Time  would  then  be  provided  for  each  of  three  small  collaborative  groups  to  meet  over  the  system 
to  work  on  a web  development  project.  Finally  a technology  product  or  skill  might  be  demonstrated  with  a lab 
activity  to  follow. 

Outside  of  class  students  were  expected  to  interact  on  TWISTER  at  least  weekly.  Weekly  TWISTER 
activities  included  readings  discussions,  posting  internet  finds,  and  responding  to  weekly  scenarios  developed  by  the 
instructional  team.  It  was  envisioned  that  TWISTER  would  also  serve  to  support  the  collaborative  groups  developing 
a topical  web-page.  Several  individual  written  activities  were  also  required  assignments. 


Data  Analysis 

The  primary  goal  of  a case  study  is  to  look  deeply  into  the  complexities  of  a single  phenomenon  or  instance. 
The  analysis  process  was  formative  as  were  some  details  of  the  overall  research  procedures.  Initially,  the  data  were 
approached  openly  with  only  the  initial  research  questions  in  mind.  Following  one  open  coding  review,  the  data  were 
searched  categorically  in  the  NUD*IST  database.  Using  the  reports  generated  with  these  searches,  profiles  of 
participants,  the  learning  environment,  and  the  communication  and  interaction  among  participants  were  developed. 
Finally,  assertions  from  these  profiles  were  categorized  and  validated  and  synthesized  using  relevant  literature. 

Initially,  transcribed  data,  along  with  the  computer  transcripts,  were  uploaded  into  NUD*IST  software  to 
assist  in  data  management  and  thematic  searching.  It  also  served  as  a verification  for  more  consistent  coding  of  the 
participants’  own  words  than  what  coding  by  hand  might  have  produced. 

Profiles  of  each  participant  were  developed  including  both  students  and  the  instructional  team.  Member 
checking  was  also  done  with  student  participant  profiles  by  emailing  each  student  participant  a draft  of  their  own 
profile  and  requesting  comments. 

Using  research  questions  1 and  2 as  guides,  data  sources  were  reviewed  to  develop  a profile  of  the  learning 
environment  including  utilization  of  TWISTER.  Results  comprised  of  data  from  across  all  data  sources  were  sorted 
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according  to  one  of  the  following  aspects  of  the  learning  environment:  course  structure  and  design,  course  activity, 
physical  facilities,  technology  implementation  and  technology  utilization  by  both  students  and  the  instructional  team. 

The  last  review  of  the  data  examined  the  communication  and  interaction  among  the  participants  in  the 
course.  Again,  reports  were  generated  using  keyword  searches  in  the  NUD*IST  database.  These  reports  were  most 
useful  in  understanding  student  and  instructional  team  perceptions  of  interaction  and  communication  in  each  of  the 
three  learning  environments:  face-to-face,  compressed  video,  and  TWISTER.  Additionally,  the  TWISTER 
transcripts  were  examined  closely  to  examine  participation,  types  of  communication,  and  levels  of  interaction  of 
individual  students  for  both  the  planned  and  unplanned  online  activities.  Using  data  summaries  of  each  small 
collaborative  group  project,  cross  case  comparisons  were  also  completed. 

Validity 

All  of  the  data  collected  came  from  the  participants.  This  approach  lends  itself  to  an  emic  data  analysis 
process.  As  data  were  analyzed,  interpreted,  and  prepared  for  presentation,  I made  a rigorous  effort  to  include  the 
participant’s  own  words  (Patton,  1990).  Although  I am  a participant,  I am  also  a researcher;  in  some  instances  and 
for  some  research  questions,  I relied  on  my  own  typologies  or  categories  in  order  to  interpret  the  data.  In  these 
instances  I have  used  what  Patton  (1990)  calls  an  “analyst-constructed”  or  etic  approach  to  data  analysis. 

Because  I was  involved  in  the  research  as  both  a researcher  and  a participant,  it  was  impossible  to 
completely  separate  my  own  interpretations  from  the  reality  of  each  of  the  participants.  In  order  to  guard  against  my 
own  interpretations  dominating,  planned  member  checking,  triangulation,  and  reminders  to  myself  of  my  own 
worldview  and  potential  biases  were  integrated  into  the  analysis  plan.  Finally,  all  assertions  are  discussed  in 
comparison  to  relevant  literature. 

Reflection  on  the  Methodology 

This  methodology  offered  a unique  and  in-depth  understanding  of  one  class’  experience  using  a groupware 
tool  to  support  course  activities.  What  makes  this  study  unique  is  that  the  perspectives  of  all  the  participants  were 
studied  in-depth.  Gaining  the  learner’s  perspective  not  only  informed  the  researcher’s  understanding  of  this  case,  but 
also  offered  learners  the  opportunity  for  reflection  on  all  aspects  of  the  course.  This  study  attempted  to  contribute  to 
the  body  of  knowledge  in  instructional  technology  by  identifying  the  issues  involved  in  studying  distance  learning 
and  groupware  environments  (Koschmann,  1995;  Mclsaac  & Gunawardena,  1996). 

Nonetheless,  there  were  clearly  limitations  that  have  been  illuminated  in  retrospect.  The  most  dominating 
limitation  was  the  relationship  between  the  instructor  and  the  researcher.  The  instructor  and  researcher  were  involved 
in  the  design  of  the  course  and  the  design  of  the  research.  Each  having  a hand  in  both  presented  several  problems. 
These  problems  emerged  during  data  collection,  data  analysis,  research  design,  and  course  implementation.  Both 
learners’  experiences  and  the  instructional  team  members’  experiences  were  affected. 

First,  the  instructor  of  the  course  was  required  to  wear  ’’multiple  hats.”  She  was  continuously  asked  to  be 
conscious  of  her  roles  as  research  adviser  and  course  instructor.  This  consciousness  of  dual  roles  by  the  instructional 
team  impacted  data  collection. 

There  was  hesitancy  among  instructional  team  members  to  discuss  particular  students,  more  so  in  a 
potentially  negative  light,  and  have  it  captured  on  tape.  For  example,  members  of  the  instructional  team  would  ask 
for  the  tape  to  be  turned  off,  if  the  conversation  was  moving  in  that  direction.  Therefore,  these  discussions  were  not 
captured  and  raw  data  was  sometimes  lost.  As  a result  some  potential  findings  could  not  be  substantiated.  The 
findings,  primarily  regarding  the  challenges  that  emerged  in  the  course,  surrounded  issues  of  instructional  design  and 
technology  implementation.  This  captured  only  two  parts  of  the  picture.  A third  part  was  the  participants' 
responsibility  for  meeting  expectations  in  the  course  and  making  use  of  the  available  tools.  Although  the  study  was 
designed  to  capture  learner  characteristics  that  might  influence  participation  and  communication,  this  failed  for  two 
reasons.  First,  the  data  collection  instruments  gave  the  learners  an  opportunity  to  describe  challenges  they  faced  in 
the  class,  but  did  not  ask  them  to  reflect  on  what  their  role  may  have  been.  Second,  follow-up  data  collection 
opportunities  were  not  planned  to  gain  participants’  reflections  after  this  issue  was  discovered  late  in  the  data  analysis 
phase.  Moreover,  it  was  the  tendency  of  the  instructional  team  to  take  responsibility  for  problems  as  issues  of 
instructional  design  and  limitations  of  the  technology.  This  is  what  was  captured  in  the  data  and  thus  presented  as 
findings. 

Because  of  these  "multiple  hats"  learner  perceptions  were  also  affected.  Data  suggested  that  the  instructor 
also  carrying  a research  adviser  role  caused  some  mistrust  between  learners  and  the  instructor.  Moreover,  learners 
perceived  the  researcher  as  closely  and  personally  tied  to  the  groupware.  There  was  some  concern  as  to  how  that 
might  have  impacted  the  data  and  findings. 

Research  also  had  an  impact  on  the  selection  of  a groupware  product.  In  order  to  collect  transcripts  of  the 
online  discussions,  the  product  had  to  maintain  a record  of  the  communication.  TCB  Works  offered  the  opportunity 
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to  customize  the  usage  privileges  of  participants.  For  this  study  users  were  not  able  to  delete,  move,  or  modify  the 
transcripts.  Several  users  found  this  lack  of  capabilities  to  be  frustrating. 

Finally,  this  feeling  of  mistrust  was  evident  during  data  collection.  The  researcher  sensed  during  individual 
interviews  that  students  were  not  completely  honest  regarding  their  experience  in  the  course.  This  was  confirmed  for 
one  participant.  In  most  of  the  data,  she  described  an  overall  positive  experience  in  the  course,  however  on  her  post- 
survey she  revealed  somewhat  negative  attitudes  and  experiences. 

These  limitations  served  as  a filter  for  presenting  and  validating  the  conclusions  and  recommendations 
asserted  in  the  next  sections.  Despite  the  recognized  difficulties,  important  outcomes  emerged. 

Discussion  of  the  Findings 

As  Stake  (1995)  suggests,  meaning  is  added  to  data  after  breaking  apart  the  instance  and  putting  it  back 
together.  Data  were  reviewed  three  times  from  the  frame  of  each  research  question.  After  completing  these  three  in 
depth  reviews,  synthesis  occurred  by  examining  the  findings  from  the  original  purpose  of  the  research.  Findings  were 
sorted  according  to  the  broad  categories:  the  compressed  video  environment,  achieving  a student-centered  distance 
learning  environment,  and  integrating  the  groupware  tool.  Stepping  back  to  this  broader  frame  enabled  a threading  of 
the  findings  for  representation  and  discussion.  Conclusions  were  validated  using  relevant  literature. 

Teaching  and  Learning  in  a Compressed  Video  Environment 

Learner  participants  in  this  case  described  being  inhibited  in  interacting  over  the  compressed  video  system 
with  some  feeling  as  though  they  were  "interrupting"  in  order  to  participate.  Students  also  noted  difficulties  in  getting 
others  to  interact  with  them.  One  participant  described  her  experience  leading  a discussion  as  "pulling  teeth"  to  get 
others  to  participate.  Another  equated  her  experience  with  receiving  the  instruction  over  compressed  video  when  the 
instructor  was  not  at  her  site  to  "watching  TV"  making  it  difficult  to  pay  attention  for  longer  than  15  minutes  at  a 
time.  Several  participants  observed  that  the  subtleties  of  face-to-face  communication,  i.e.  body  language,  facial 
expressions,  no  delayed  responses,  were  lost  when  communicating  over  the  compressed  video  system.  The  instructor 
also  observed  the  student-student  interactions  as  being  low  level.  She  considered  class  readings  discussions  to  be 
"boring"  and  in  comparing  a face-to-face  class  meeting  to  one  over  the  compressed  video  system,  she  refers  to  the 
discussions  during  the  face-to-face  "like  a real  class."  Mediated  communication  through  compressed  video  impacted 
the  interactivity  among  students  as  well  as  the  interaction  between  the  students  and  instructor.  Similar  accounts  of 
these  types  of  challenges  were  reported  in  much  of  the  recent  distance  learning  research  on  communication  in 
compressed  video  environments  (Comeaux,  1995;  Fulford  & Zhang,  1994;  Kearsley,  1995;  McHenry  & Bozik, 

1995;  McHenry  & Bozik,  1997;  Mclsaac  & Gunawardena,  1996;  Repman  & Logan;  1996;  Schrum,  1996;  Tiene, 
1997;  Wolcott,  1996). 

The  interaction  between  students  and  the  instructor  took  many  forms  as  suggested  by  Garrison  and  Shale 
(1990).  There  were  the  expected  challenges  for  the  instructor  with  presenting  information  and  providing  feedback  to 
the  learners.  Additionally,  extra  time  was  needed  for  redesigning  the  activities  and  presentation  of  information  took 
time  to  redesign  for  this  environment.  Providing  feedback  to  students  required  coordination  to  get  the  right 
assignments  to  the  right  location  on  the  same  day.  In  some  instances  the  feedback  was  provided  via  supplemental 
technologies  like  fax,  phone,  or  email.  The  facilitation  of  these  types  of  responsibilities  was  distributed  between  all 
members  of  the  instructional  team.  One  issue  students  commented  on  as  being  problematic,  however,  was  that  the 
professor  was  only  available  every  other  week  for  face-to-face  communication  during  class.  The  attitude  and 
expectations  of  students  as  well  as  their  physical  location  were  key  here.  Students  at  the  local  site  were  less  likely  to 
be  accommodating  to  a physically  distant  instructor  than  were  the  remote  site  students.  Since  the  remote  site  students 
were  all  aware  that  the  course  was  being  delivered  over  compressed  video  and  because  they  gained  the  conveniences 
afforded  through  receiving  the  distance  delivery,  interacting  with  the  instructor  every  other  week  was  perceived 
positively.  For  students  at  the  local  site,  however,  it  was  perceived  less  positively.  Prior  to  the  first  night  of  class, 
many  of  the  local  site  students  did  not  realize  the  course  was  to  be  delivered  at  a distance.  Several  commented  that  it 
was  enough  to  "put  up"  with  the  videoconference  technology,  but  to  have  also  lost  access  to  the  instructor  without 
the  added  conveniences,  was  frustrating.  Students  from  both  sites  agreed  that  the  supplemental  technologies  did  not 
provide  the  immediate,  two-way  interaction  sometimes  necessary. 

"Clarity"  in  communication  also  emerged  as  a major  issue  in  this  class  with  several  students.  For  example, 
there  was  a request  for  clarification  of  the  assignments  on  almost  a weekly  basis.  This  information  was  provided  in 
the  syllabus  and  discussed  during  class,  yet  there  was  still  confusion  among  some  participants.  Other  instances  of  a 
lack  of  clarity  included  students  not  being  prepared  for  class  because  they  "didn’t  know"  or  did  not  remember 
information  disseminated  during  a class  session. 

Many  of  the  above  challenges  have  to  do  with  what  researchers  have  referred  to  as  psychological  distance 
(Comeaux,  1995;  Wolcott,  1994).  Being  aware  of  this  from  the  accounts  described  in  the  literature,  the  instructional 
team  designed  the  course  to  incorporate  instructional  strategies,  techniques,  and  activities  that  would  minimize  the 
psychological  distance  therefore  creating  a more  studenLcentered  environment. 
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Designing  Student-centered  Instructional  Strategies  for  Distance  Learning  Environments 

Using  a wide  variety  of  activities  in-class  and  for  individual  assignments  was  an  attempt  to  cater  to  varying 
knowledge  and  skill  levels,  learning  styles,  and  location,  and  to  make  the  most  of  the  technologies  employed.  As 
suggested  in  the  literature  most  worked  well  in  the  environment.  Rapport  building  strategies  included  icebreakers, 
being  available  before  and  after  class,  and  building  in  time  for  questions.  Interaction  enhancing  strategies  included 
using  a variety  of  activities  beside  lecture,  encouraging  participation  from  each  site,  and  making  a multitude  of  ways 
available  to  access  other  participants  (Schrum,  1996;  Wolcott,  1996).  The  following  two  strategies  yielded  rich 
findings  and  were  explored  in  more  detail:  open  learning  activities  and  collaborative  activities. 

The  instructor  had  always  used  open  learning  activities  in  her  course  in  order  to  provide  learners  with  an 
opportunity  to  make  the  learning  authentic  and  meaningful  for  themselves.  In  reflection,  the  instructor  admitted  that 
there  are  always  some  students  who  become  uncomfortable  with  defining  their  own  learning  goals.  Often  students 
have  no  experience  defining  their  goals  and  negotiating  them  with  an  instructor.  Similarly,  in  this  case  there  were 
some  students  who  were  uncomfortable  with  this  process.  Eventually  all  students  except  one  found  these  open-ended 
assignments  to  be  personally  meaningful.  Given  these  results,  this  strategy  proved  successful  for  this  case. 

Collaborative  activities  were  used  as  a means  of  community  building  to  bridge  the  psychological  distance 
and  increase  interaction.  Collaborative  learning  strategies  provide  opportunities  for  learners  to  build  knowledge 
though  discourse  by  solving  a problem  or  completing  a task  in  a group  (Blumenfeld,  Marx,  Soloway,  & Grajcik, 
1996;  Joyce  & Weil,  1992).  Although  collaborative  learning  activities  are  known  to  be  difficult  to  implement  in 
distance  learning  environments  because  of  the  geographic  and  temporal  separation  of  the  learners,  if  carefully 
designed  they  are  possible  (Henri  & Rigault,  1991).  In  this  case  there  were  several  opportunities  designed  for 
learners  to  collaborate  in  small  groups  and  dyads  across  sites.  Findings  specifically  related  to  collaborative  activities 
supported  by  TWISTER  are  presented  in  the  next  section. 

Using  characterizations  of  collaborative  work  from  the  literature,  the  small  group  projects  were  more 
successful  in  providing  students  the  opportunity  to  work  closely  with  others  and  learn  by  completing  a task  (Barnes 
& Todd,  1995,  Blumenfeld,  Marx,  Soloway,  & Grajcik,  1996;  Crook,  1994,  McReary,  1990;  Sharpies,  1993;  Webb, 
1992).  As  seen  in  the  Readings  Dyads  little,  if  any,  collaboration  in  preparing  the  readings  occurred  for  most  of  the 
dyads. 

Two  of  the  three  groups  experienced  problem  behaviors  as  identified  by  researchers  in  collaborative 
learning  (Blumenfeld,  Marx,  Soloway,  & Grajcik,  1996;  Johnson  & Johnson,  1992).  One  group  experienced 
problems  with  accountability.  A sole  member  participating  in  the  class  from  the  remote  site  did  not  play  a significant 
role  in  her  group's  final  product.  In  another  group,  there  was  a rich-get-richer-effect.  Because  one  participant  had 
web  development  experience,  she  took  over  the  responsibility  for  this  and  neither  of  the  other  two  participants  gained 
skills  in  this  area  as  a result  of  the  activity,  yet  she  gained  more  experience.  As  seen  in  both  these  identified 
collaboration  problems  there  was  a link  between  the  participant  with  the  problem  behavior  and  their  physical 
location.  The  participant  with  web  development  expertise  did  not  share  her  web  development  skills  with  the  other 
members  of  her  group  because  they  were  not  physically  accessible  during  the  times  she  did  the  development.  The 
participant  accused  of  low  participation  was  a self-described  introvert  and  low  level  technical  skill  holder.  She 
claimed  she  had  difficulty  contributing  to  her  group  especially  since  she  was  the  only  member  at  the  remote  site.  This 
physical  separation  of  one  learner  from  the  other  two  impacted  the  cohesion  of  these  two  groups.  The  small  number 
of  participants  in  the  cross-site  group  was  a likely  factor  as  well. 

Incorporating  a CSCL  Tool  in  a Distance  Learning  Environment 

In  order  to  facilitate  many  of  these  student-centered  strategies,  TWISTER  was  devised  as  an  extension  of 
the  learning  environment.  Findings  regarding  its  implementation,  use,  and  support  for  the  activities  follow. 

The  instructional  team  envisioned  and  designed  the  learning  environment  to  include  use  of  TWISTER  to 
support  a range  of  instructional  strategies.  Although  referred  to  as  a supplement  and  never  used  solely  as  a means  to 
support  an  instructional  activity,  the  team  planned  for  TWISTER  to  be  an  integral  part  of  the  course.  Despite  this 
intention,  neither  the  instructor  nor  the  students  used  it  in  this  way.  Difficulties  encountered  with  the  features  and 
functionality  of  TWISTER  proved  to  limit  the  mode  of  implementation  to  a less  integral  mode,  i.e.  what  Harasim 
(1996)  would  refer  to  as  adjunct.  Most  participants  found  little  value  in  the  added  features  of  having  an  integrated 
course  support  tool.  The  value  did  not  outweigh  the  challenges  they  identified  with  the  features  and  functionality. 

The  design  of  TWISTER  was  based  on  findings  from  the  literature  and  pilot  studies  with  an  earlier 
prototype  (Ahem,  1993;  CGTV,  1992;  Guzdial,  Turns,  Rappin,  & Carlson,  1995;  Kaplan  & Carol,  1995;  Lajoie, 
1993;  Lebow,  Wager,  Marks,  & Gilbert,  1996;  Scardamalia  & Bereiter,  1994;  Schrum,  Fitzgerald,  & Luetkehans, 
1997;  Suthers  & Weiner,  1995;  Wan  & Johnson,  1994).  Desired  features  of  the  tool  included  space  to  post 
information  or  orienting  scenarios,  asynchronous  and  synchronous  capabilities,  support  for  threaded  discussions, 
hypertext  capabilities,  information  searching  capabilities,  scaffolding  for  metacognitive  process  structures,  and  the 
capability  for  participants  to  produce,  share,  and  display  products.  In  this  case,  the  prototype  tool,  TWISTER,  did 


not  incorporate  scaffolding  or  production  tools.  Moreover,  as  discussed  in  the  following  section  the  synchronous 
capabilities  had  limitations  in  functionality  impacting  use  in  this  class. 

Synchronous/ Asynchronous.  Considerations  Schrum  (in  press)  suggests  for  when  to  choose  synchronous 
tools  include  number  of  participants,  skills  of  learners,  and  the  type  of  task.  Although,  there  was  only  one  trial  of  a 
synchronous  session  by  the  participants,  that  one  instance  demonstrated  TWISTER’S  weak  synchronous 
conferencing  capabilities.  Learners  in  one  of  the  small  work  groups  were  disappointed  by  time  lags  and  access 
problems  during  their  attempt. 

In  this  study,  several  types  of  tasks  were  designed  to  examine  how  the  different  capabilities  of  each  learning 
environment  support  the  varying  needs.  Types  of  tasks  built  in  were  brainstorming  and  idea  generation  (Scenarios), 
information  sharing  (Internet  Finds),  dyadic  negotiations  (Readings  discussions),  and  small  group  projects  which 
incorporated  a range  of  tasks  including  brainstorming,  problem  solving,  and  construction/production.  Harasim 
(1993)  also  recommends  task  type  as  an  important  consideration  for  choosing  synchronous  versus  asynchronous 
conferencing  tools. 

Since  there  was  such  little  use  of  TWISTER  it  would  be  unfounded  for  this  researcher  to  make  any 
inferences  for  best  uses  of  a groupware  tool  with  synchronous  and  asynchronous  conferencing.  However,  a 
comparison  of  observations  of  the  asynchronous  use  of  TWISTER  with  the  synchronous  use  of  compressed  video  for 
groups  to  come  to  a consensus  indicated  that  decision-making  was  difficult  in  asynchronous  online  environments. 
Instead  of  breaking  tasks  to  the  level  of  specificity  described  above,  Harasim  (1993)  looked  at  tasks  as  being 
convergent  and  divergent.  As  seen  in  the  small  work  groups,  compressed  video  (synchronous)  provided  a better 
environment  for  some  types  of  tasks.  When  the  groups  described  their  group  process,  all  of  them  stated  that 
compressed  video  was  the  better  means  for  decision-making,  a convergent  activity.  Data  were  also  consistent  with 
Harasim’s  (1993)  findings  that  asynchronous  tools  are  better  for  divergent  activities,  e.g.  brainstorming,  discussion, 
information  sharing. 

Hvpertext/threaded  discussions.  Two  students  recognized  the  value  of  being  able  to  structure  and  organize 
a conference,  i.e.  a threaded  discussion.  Although  these  two  students  attempted  to  do  so  using  the  hierarchical 
structure,  other  participants  made  little  use  of  the  organizing  capabilities.  Data  showed  that  there  was  little 
interaction  reaching  the  level  of  a dialogue  suggesting  that  learners  did  not  have  a need  to  impose  an  organizational 
structure  on  their  postings. 

Space  for  macrocontext.  The  hierarchical  structure  of  TCB Works  worked  well  in  serving  as  a place  to 
present  orienting  scenarios.  The  weekly  scenarios  proved  to  be  the  favored  use  of  TWISTER  by  both  the  learners 
and  instructional  team.  With  each  weekly  scenario  given  a dedicated  space,  learners  could  post  responses  and  follow 
the  threaded  postings.  Although  the  responses  to  the  weekly  scenarios  never  reached  the  level  of  a dialogue,  students 
had  no  complaints  or  problems  using  the  tool  for  scenarios. 

Other  features  not  utilized  by  participants.  Two  other  features  available  to  users  of  TWISTER  were 
information  searching  and  voting.  The  information  searching  feature  allowed  participants  to  search  the  world  wide 
web  while  communicating  in  TWISTER.  Although  this  feature  was  introduced  and  recommended  to  learners  and 
identified  as  a need  by  at  least  one  small  work  group,  the  feature  was  never  employed.  Participants  claimed  they  did 
not  realize  it  was  available.  The  second  feature  not  used  by  participants  was  the  voting  feature  built  into  TCBWorks, 
the  groupware  product.  This  function  was  not  particularly  intuitive,  and  it  was  likely  participants  would  have  needed 
training.  The  voting  feature  was  there  to  assist  groups  of  users  in  coming  to  consensus.  Although  there  was  an 
expressed  need  for  support  in  decision-making  by  the  small  groups,  the  voting  feature  did  not  appear  to  translate  well 
to  a learning  situation.  The  instructional  team  mentioned  voting  as  an  available  feature  when  TWISTER  was 
introduced,  but  they  did  not  encourage  its  use  nor  provide  specific  training. 

Learner-interface  Interaction 

Hillman,  Willis  and  Gunawardena  (1994)  suggest  an  additional  type  of  interaction  experienced  in  distance 
learning  environments,  learner- interface  interaction.  This  type  of  interaction  refers  to  the  ’'transparency”  of  the 
technology.  TWISTER  was  a prototype  groupware  tool,  and  although  it  was  tested,  was  still  laden  with  design  and 
functionality  difficulties.  Data  suggested  that  these  peculiarities  kept  the  tool  from  becoming  transparent  for  these 
learners. 

TWISTER’S  limitations  in  functionality  presented  challenges  to  learners  interacting  with  the  tool. 
Additionally,  learners  perceived  as  an  inconvenience  having  to  "go  to  it"  in  order  to  interact.  Conferencing  systems 
like  TWISTER  are  often  referred  to  as  pull  technologies  (Dennis,  Pootheri,  & Natarajan,  1997).  Email  can  be 
described  as  a push  technology,  the  opposite  of  a pull  technology.  With  push  technologies  like  email  and  Listservs, 
conferencing  information  and  messages  are  sent  to  participants’  email  box.  If  they  are  regular  email  users,  as  many 
were  in  this  class,  checking  email  was  already  a familiar  and  often  daily  activity.  Knowing  this,  learners  and 
instructors  often  opted  for  email  when  they  wanted  to  be  sure  the  information  would  be  received  or  if  they  wanted  to 
initiate  a two-way  dialogue. 


Learner  Characteristics 

Consistent  with  studies  examining  online  learning  environments,  data  in  this  case  indicated  that  participant 
characteristics  proved  to  be  indicators  for  how  and  when  they  used  the  groupware  tool  (Eastmond,  1995;  Hiltz,  1990; 
Schrum,  1996).  Characteristics  emerging  as  important  included:  access,  experience  and  skill  level,  and  attitude.  The 
type  and  means  of  access  a student  had  to  the  internet  outside  of  class  time  affected  their  use  of  the  groupware  tool. 
Access  was  a factor  deemed  important  at  the  outset  of  the  research,  however,  the  full  extent  of  its  impact  was 
underestimated.  Not  only  did  learners  with  unreliable  or  no  access,  struggle,  but  learners  with  limited  access  also 
encountered  difficulties.  Learners  with  home  or  work  access,  only,  were  sometimes  limited  in  the  days  or  time  of  day 
they  could  interact  on  the  system.  Furthermore,  those  with  limited,  unreliable  or  no  access  also  affected  the  work  of 
others  trying  to  collaborate  with  them. 

Previous  experience  and  technology  skill  level  were  important  indicators,  too.  This  was  seen  from  both  ends 
of  the  continuum  including  low  skills  to  very  high  skills  and  previous  experiences,  both  positive  and  negative. 

Several  studies  have  identified  similar  characteristics  of  readiness  for  participating  in  online  environments 
(Eastmond,  1995;  Hollingshead,  McGrath,  & O’Connor,  1993).  In  order  to  expect  learners  to  participate,  a minimal 
skill  level  was  required  from  the  outset.  Again,  in  collaborative  environments,  one  learner’s  struggle  can  impact  a 
whole  group  or  even  class.  But  still,  some  students  bring  with  them  notions  and  habits  with  which  they  are 
accustomed.  This,  too,  can  affect  their  participation. 

Finally,  attitudes  toward  several  aspects  of  the  learning  situation  were  apparent  indicators  for  participation 
and  utilization.  Consistent  with  what  O’Malley  and  Scanlon  (1990)  found,  despite  learners’  range  of  attitudes  toward 
group  work,  most  described  their  experience  with  the  groupwork  in  this  course  as  positive.  Findings  from  this  case 
differ  from  O'Malley  and  Scanlon  (1990)  however,  with  the  small  work  groups.  Some  learners  expressed  negative 
opinions  about  having  participated  in  cross-site  collaborative  groups  and  most  learners  agreed  the  groupware  tool  did 
not  support  their  small  group’s  work  well. 

This  study  examined  two  types  of  interaction  using  the  groupware  tool:  learner-learner  interaction  and 
learner-instructor  interaction.  Participants  in  this  class  used  the  tool  to  share  ideas  and  information  and  to  maintain 
records.  There  was  little  demonstration  of  interaction  reaching  beyond  the  level  of  individuals  posting  single 
messages  or  pieces  of  information  to  the  large  group.  This  finding  followed  a one-to-many  pattern  of  interaction 
proposed  by  Romiszowski  and  Mason  (1996)  as  common  to  asynchronous  tools.  Data  suggested  that  utilization  of 
groupware  tools  is  a complex  and  multifaceted  issue.  The  limited  use  in  this  case  was  the  result  of  problems  learners 
had  with  system  implementation,  system  access,  as  well  as  the  features  and  the  functionality  of  TWISTER. 

Recommendations  for  Practice 

These  recommendations  for  practice  result  from  a process  of  naturalistic  generalization.  Through  careful 
data  analysis,  interpretation,  and  audit  trailing,  the  findings  from  this  case  study  were  examined  in  light  of  the  related 
literature.  Caution  was  taken  given  the  number  of  limitations  in  this  study. 

Assess  the  participants.  This  case  demonstrated  the  impact  that  learner  knowledge,  skills,  experience,  and 
attitude  can  have  on  the  learning  environment.  Before  a new  course  begins  it  would  be  helpful  to  gain  a sense  of 
these  characteristics  for  the  particular  set  of  learners  in  the  learning  situation.  If  there  are  wide  ranges  in  skills  and 
experience,  activities  can  be  tailored  to  meet  individual  needs. 

Learners  need  to  be  prepared.  Learner  “readiness”  as  Eastmond  (1995)  suggested  is  important  to  successful 
implementation.  First,  learners  need  to  understand  how  the  course  will  be  delivered.  Additionally,  in  online  or 
groupware  environments  they  will  want  to  be  sure  they  have  sufficient  access  to  equipment. 

The  instructor  should  be  prepared  for  the  time  it  takes  to  manage  a DL  course.  In  this  instance  there  were 
three  members  on  the  instructional  team,  and  in  every  class  meeting  each  member  of  the  team  was  occupied  with 
responsibilities.  It  is  important  that  an  instructor  not  underestimate  the  amount  of  time  required. 

Employ  short  and  varied  activities  during  class  sessions.  As  one  participant  described,  her  attention  span 
averaged  aboutl5  minutes  when  viewing  instruction  over  the  compressed  video  system.  To  expect  learners  to  be  able 
to  pay  attention  for  longer  periods  of  time  is  unreasonable.  Not  only  should  activities  be  short,  but  also  varied.  By 
changing  the  type  of  activity,  learners  may  find  more  opportunities  to  participate. 

Constant  clarification  may  be  necessary.  Given  that  the  average  attention  span  in  compressed  video 
environments  may  be  only  15  minutes  long,  it  can  be  expected  that  the  learners  may  miss  some  information.  Plan  for 
this  by  reviewing  often,  or  providing  supplemental  handouts. 

Interaction  does  not  come  easily  in  DL  Environments.  Despite  the  difficulties,  low  levels  of  interaction  need 
not  be  settled  for.  Interaction  can  be  constantly  encouraged  by  incorporating  participatory,  discussion-based 
activities,  leaving  little  room  for  lecture.  High  levels  of  interaction  may  need  to  be  modeled. 

Build  community.  As  the  instructor  in  this  case  believed,  building  community  is  important  in  DL 
environments.  However,  rotating  locations  every  week  may  not  be  the  solution.  Off  campus  learners  registered  for 
the  course  with  the  knowledge  there  would  be  differences;  on  campus  students  did  not.  These  expectations  can  color 
learner  attitudes  and  influence  the  dynamics  of  the  class.  Instead  the  instructor  can  make  themselves  readily 
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accessible  in  other  ways.  For  off  campus  students,  instructors  might  consider  taking  the  initiative  to  check  in  with 
them  outside  of  class  sessions  through  supplemental  media,  e.g.  email,  telephone,  etc. 

Open  and  collaborative  activities  are  important.  Open  and  collaborative  activities  are  important  in  building 
student-centered  DL  environments.  Open  activities  give  learners  the  opportunity  to  experience  authentic  and 
meaningful  learning.  Cross-site  collaboration  helps  to  build  community  and  increase  interaction.  Consideration, 
however,  should  be  given  to  the  group  size  and  location  of  participants.  Consider  using  an  equal  number  of  learners 
at  each  site.  If  groups  are  constructed  based  on  particular  characteristics,  e.g.,  expertise,  locations  of  these 
individuals  should  be  noted.  Also  consideration  should  be  given  to  providing  supplemental  technologies  as 
necessary.  A groupware  tool  may  not  be  the  only  solution;  email,  synchronous  chat  tools,  telephone,  and  fax  could 
also  be  considered. 

Allow  sufficient  training  and  time  for  learning  groupware  software.  In  presenting  learners  with  a new 
software  tool,  ample  training  and  guidance  needs  to  be  provided.  Becoming  adept  at  using  the  groupware  tool  could 
take  longer  than  ten  weeks.  For  most  people,  groupware  would  be  a new  class  of  software.  This  learning  curve 
coupled  with  learning  the  content  of  the  course  may  require  additional  time  as  well  as  practice  before  participants 
become  proficient  at  using  it. 

Consider  the  Best  Uses  of  Tool  Features  for  Specific  Task  Types.  In  deciding  whether  to  use  a synchronous 
or  asynchronous  tool  one  should  consider  whether  the  activity  is  convergent  or  divergent  as  Harasim  (1995)  and 
Danielsen  (1994)  recommend  and  findings  in  this  study  concur.  Consideration  should  also  be  given  to  whether  the 
tool  is  a “push”  or  “pull”  technology.  Based  on  usage  in  this  case,  push  technologies  like  email  ought  to  be 
considered  for  addressing  an  individual  or  a one-to-one  dialogue.  Pull  technologies,  on  the  other  hand,  appear  to 
work  well  for  larger  groups  and  maintaining  a group  record. 

Selecting  a groupware  tool.  Selection  criteria  needs  to  begin  with  whether  the  tool  will  translate  well  to  a 
learning  environment.  As  in  this  case,  adapting  a tool  originally  developed  for  business  work  groups  posed  some 
challenges.  These  challenges  emerged  because  the  tool  itself  had  to  be  adapted  to  fit  the  intended  uses  and  not  all 
the  features  were  usable.  Also,  the  tool  selected  has  to  be  fully  functional.  Using  a prototype  became  an  issue  for  this 
group.  “Quirks”  and  “kinks”  were  eventually  accommodated,  but  neither  easily  nor  without  frustration.  The  interface 
is  also  crucial.  Learners  should  not  be  expected  to  conform  to  the  tool.  Again  eliminating  frustration  and  difficulties 
up  front  can  help  build  a positive  attitude  in  users.  Finally,  the  value  added  versus  required  effort  in  selecting  a tool 
needs  to  be  carefully  weighed.  In  this  case,  there  was  too  little  gained  as  a result  of  exerting  a lot  of  effort  to  use 
TWISTER. 

Recommendations  for  Further  Study 

Future  studies  considering  similar  overall  questions  to  those  in  this  study  would  be  remiss  not  to  be  certain 
that  the  groupware  tool  employed  is  fully  functional  and  meets  the  criteria  described  previously  in  the 
Recommendations  for  Practice  section.  The  tool  should  be  designed  to  support  learning  environments.  Hopefully,  it 
will  be  possible  to  employ  a tool  that  incorporated  fully  functional  features  as  described  in  the  proposed  model. 
Additional  features  would  include  scaffolding  for  instructional  strategies  and  metacognitive  processes,  and  shared 
production  tools. 

Also,  more  case  studies  similar  to  this  would  be  useful  for  cross-case  comparisons.  Similar  studies  might 
also  explore  content  areas  outside  of  instructional  technology  without  the  methodological  limitations  described  in  the 
Reflection  on  the  Methodology  section. 

Examining  ways  to  improve  distance  learning  environments  continues  to  be  an  important  line  of  inquiry. 
New  technologies  and  new  pedagogies  have  not  bridged  the  “psychological  distance”  when  participants  are 
separated  by  space  and  time.  Further  questions  to  push  this  research  area  should  consider  such  issues  as: 

• What  are  minimum,  necessary,  or  desired  levels  of  interaction? 

• What  can  be  done  about  this  issue  of  “clarity”  and  missed  information  in  DL  environments? 

• What  are  best  practices  for  employing  push  versus  pull  technologies  and  synchronous  versus  asynchronous  tools? 

• What  types  of  tasks  can  be  supported  by  the  various  technologies? 
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Summary 

In  this  case,  a great  deal  was  learned  about  using  a groupware  tool  to  supplement  a distance  learning  course 
delivered  over  a compressed  video  system.  Despite  the  careful  design  and  crafting  of  a student-centered  distance 
learning  environment,  participants  identified  a wide  range  of  challenges.  Instructors  and  students  recognized 
limitations  in  interactivity  frequency  and  levels,  some  activities  did  not  work  as  intended  and  others  were 
compromised  by  the  environment  or  participants  themselves.  The  open  and  collaborative  activities  encouraged 
rapport  building  and  enhanced  interaction.  Nonetheless,  challenges  remained. 

Use  of  TWISTER  to  assist  in  facilitating  the  student-centered  activities  also  presented  challenges.  The  tool 
did  not  support  the  activities  as  intended.  Given  the  range  of  activities,  the  low  usage  of  the  groupware  was 
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surprising.  Factors  that  influenced  participation  included  participant  characteristics  and  the  learner-interface 
interaction  with  the  tool.  A lack  of  a compelling  need  may  have  existed,  even  though  activities  were  built  around 
TWISTER  use.  As  for  the  small  groups,  it  was  evident  that  email,  despite  its  lack  of  structure,  was  the  preferred 
online  communication  medium.  For  the  learners,  the  added  value  of  the  features  afforded  by  groupware  did  not 
outweigh  the  convenience  of  a push  technology  like  email.  There  was  data  to  suggest  that  asynchronous 
communication  better  supported  divergent  tasks  while  synchronous  tools  supported  convergent  tasks. 

Based  on  these  findings,  it  appears  that  distance  learning  environments  continue  to  need  to  be  improved 
through  devising  learner-centered  activities  and  increasing  interaction.  This  can  be  accomplished  by  first  assessing 
and  preparing  the  learners  for  the  learning  environment.  Additionally,  instructional  strategies  should  include  short 
and  community  building  activities.  Finally  extra  time  should  be  planned  for  management  issues. 

Incorporating  a groupware  tool  will  require  extra  time  for  training  and  practice.  Tool  design  or  selection 
considerations  should  include  assessing  the  tool’s  functionality  in  a learning  environment  and  the  types  of  activities 
to  be  supported  with  the  tool. 

This  case  exemplified  many  of  the  potential  challenges  that  can  emerge  in  distance  and  online  learning 
environments.  As  internet  communication  technologies  mature,  perhaps  more  viable  groupware  tools  will  become 
available  and  participants  will  become  increasingly  comfortable  working  in  such  environments.  These  factors  may 
eliminate  many  of  the  difficulties  encountered  in  this  case.  Meanwhile,  in-depth  case  studies  such  as  this  are  valuable 
in  investigating  the  possibilities  of  the  present. 
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Abstract 

This  study  investigated  designer  background  characteristics  and  contextual  variables  that  were 
hypothesized  to  influence  the  incorporation  of  motivational  components  to  the  design  of  instructional 
materials  as  well  as  the  factors  that  may  affect  the  perceived  level  of  a project's  success.  A hypothesized 
model  was  developed  that  depicted  anticipated  relationships  (based  on  the  literature)  between  the  variable 
sets  on  the  perceived  level  of  success  of  a project.  Of  the  500  questionnaires  mailed  to  instructional  design 
practitioners,  201  valid  responses  were  returned.  Stepwise  multiple  linear  regression  analyses  and  path 
analysis  techniques  were  used  to  answer  the  11  research  questions  posed  for  this  study  and  test  the 
hypothesized  model.  The  revised  model  was  developed  from  the  results  of  stepwise  regression  analyses  and 
path  analysis  techniques.  The  study  found  that  certain  background  and  contextual  variable  influence  the 
incorporation  of  motivation  design  components  and  that  certain  factors  in  the  hypothesized  model  have  a 
significant  relationship  to  the  perceived  level  of  success  of  a project. 

Problem  Statement 

Is  motivation  an  important  component  to  the  design  of  instructional  materials?  Most  theorists  and 
practitioners  would  agree  that  motivation  should  be  a part  of  every  design.  However,  scholars  note  that  research  on 
learner  motivation  has  lagged  far  behind  research  on  the  learner’s  cognitive  characteristics  (Herndon,  1987;  Klein  & 
Keller,  1990;  Spitzer,  1996).  Traditionally  the  design  process  have  focused  on  analysis  of  the  learner’s  cognitive 
characteristics.  According  to  Richey,  (1992)  “learner  interests  are  the  most  common  attitudes  addressed  in  ISD 
models”  (p.  1 16).  Much  in  the  literature  indicates  that  while  people  accept  that  motivation  is  a desirable  aspect  of 
instruction  it  is  considered  too  vague  and  complex  a concept  to  be  applied  to  systematic  instructional  design  (Farmer, 
1989;  Keller,  1987c;  Wlodkowski,  1981). 

This  study  investigated  designer  background  characteristics  and  contextual  variables  that  were  hypothesized 
to  influence  the  incorporation  of  motivational  components  to  the  design  of  instructional  materials  as  well  as  the 
factors  that  may  affect  the  perceived  level  of  a project’s  success.  However,  the  emphasis  in  the  literature  regarding 
background  characteristics  and  contextual  variables  is  primarily  concerned  with  the  influence  each  may  have  on  the 
effectiveness  of  the  instruction  for  the  learners.  There  is  little  research  related  to  the  possible  influences  of 
background  characteristics,  attitudes,  and  use  of  instructional  design  elements  on  a designer’s  incorporation  of 
motivational  strategies  in  to  the  materials  and  on  his  or  her  perceived  success  of  a project. 

Traditional  Instructional  Systems  Design  (ISD)  models  generally  follow  a systematic  approach  to  design 
and  are  based  on  learning  theory.  They  usually  prescribe  conducting  a learner  analysis  to  determine  knowledge  and 
skills  during  the  definition  or  analysis  phase.  Learner  characteristics  such  as  age  (Bohlin,  Milheim,  & Viechnicki, 
1990;  Bohlin  & Milheim,  1994;  Richey,  1992;  Wlodkowski,  1985);  gender  (Binns  & Branch,  1994;  Canada  & 
Brusca,  1991),  culture,  ethnicity  (Eastman  & Smith,  1991;  Powell,  1997),  organizational  context  (Tessmer  & Richey, 
1997)  are  considered  important  elements  of  a thorough  audience  analysis.  Theorists  and  practitioners  agree  that 
background  characteristics  may  have  a significant  effect  on  learner  motivation  or  learning  itself.  However,  analysis 
of  designers’  knowledge  and  skills  is  a relatively  new  subject  in  the  literature.  Little  is  said  about  how  background 
and  contextual  factors  may  affect  designers  as  they  plan  instruction  and  determine  instructional  and  motivational 
strategies. 

Keller  (1983b)  defined  motivation  as  “the  magnitude  and  direction  of  behavior.  ...  it  refers  to  the  choices 
people  make  ...  and  the  degree  of  effort  they  will  exert.  Motivation  is  influenced  by  a myriad  of  internal  and  external 
characteristics  (p.  389).  He  (Keller,  1987a)  developed  the  ARCS  Model  of  Motivational  Design  from  his  theoretical 
model  (1979),  which  was  based  on  accepted  theories  of  human  motivation.  ARCS  is  an  acronym  for  attention, 
relevance,  confidence,  and  satisfaction,  the  four  components  of  the  model.  Keller  (1987b)  defined  each  component 
as  follows: 


• Attention  --  Capturing  the  interest  of  learners;  stimulating  the  curiosity  to  leam. 

• Relevance  - Meeting  the  personal  need/goals  of  the  learner  to  affect  a positive  attitude. 

• Confidence  - Helping  the  learners  believe/feel  that  they  will  succeed  and  control  their  success. 

• Satisfaction-Reinforcing  accomplishment  with  rewards  (internal  and  external)  (p.2) 
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Methods  and  Procedures 


A hypothesized  model  was  developed,  based  on  a review  of  the  literature,  that  identified  variables  and  their 
subcomponents  and  the  possible  relationships  between  and  among  them.  The  variables  sets  used  in  the  hypothesized 
model  were  (a)  designer  background,  (b)  instructional  context,  (c)  attitude  toward  motivational  design,  (d)  use  of 
ISD  components,  (e)  incorporation  of  motivation  design  components,  and  (f)  perceived  level  of  the  project’s  success. 
Each  of  these  sets  had  subcomponents,  which  constituted  the  variables.  The  primary  hypothesis  was  that  one  or  more 
of  these  variables  would  have  a correlative  relationship  with  the  designer’s  application  of  motivational  components 
of  attention;  relevance,  confidence,  and  satisfaction  to  the  design  process.  A second  hypothesis  was  that 
incorporation  of  motivation  would  be  shown  to  have  an  impact  on  the  perceived  success  of  the  project.  (See  Figure 
1) 

The  survey  instrument  was  an  81 -item  questionnaire  with  five  sections  mapped  to  the  variable  sets  in  the 
hypothesized  model  was  designed  and  developed  by  the  researcher.  Before  finalizing  the  questionnaire,  it  was  given 
to  25  members  of  the  Michigan  Chapter  of  ISPI  (International  Society  for  Performance  Improvement)  The 
questionnaire  was  mailed  to  500  members  of  ISPI  throughout  the  50  United  States  and  Puerto  Rico.  There  were  201 
usable  questionnaires,  which  constituted  a 40%  response  rate.  Descriptive  statistics  were  reported  to  provide  a 
profile  of  the  respondents. 

The  characteristics  in  201  respondents  included  the  following:  a fairly  even  representation  of  men  (n=91, 
45%)  and  women  (n=l  10, 56%).  Nearly  all  of  the  participants  were  Caucasian  (n=186,  93%)  with  few  (n=15,  7%) 
representing  other  ethnic  groups.  Most  respondents  (n=145,  72%)  had  a master’s  degree  or  above.  The  majority  of 
the  sample  (n=  130,  65%)  were  more  than  40  years  old,  and  had  an  average  of  13  years  of  design  experience.  Nearly 
all  respondents  (n=156,  79%)  rated  their  design  expertise  as  high  or  very  high,  but  only  76  (38.5%)  claimed  to  have 
high  or  very  high  expertise  in  motivational  design. 

The  data  analyses  answered  1 1 research  questions.  The  hypothesized  model  (See  Figure  1)  depicted 
expected  relationships  between  the  following  variables: 

1 . Designer  background  and  attitude  toward  motivation. 

2.  Instructional  context  and  attitude  toward  motivation. 

3.  Designer  background  and  use  of  cores  ISD  elements. 

4.  Instructional  context  and  use  of  core  ISD  elements. 

5.  Designer  background  and  incorporation  of  motivation  design  components,  (analysis,  attention,  relevance,  confidence,  and 
satisfaction). 

6.  Instructional  context  and  the  incorporation  of  motivation  design  components,  (analysis,  attention,  relevance,  confidence, 
and  satisfaction). 

7.  Designer  attitudes  toward  motivation  and  the  incorporation  of  motivation  and  its  components,  (analysis,  attention, 
relevance,  confidence,  and  satisfaction).  1 

8.  Designer  use  of  core  ISD  elements  and  the  incorporation  and  the  incoiporation  of  motivation  components,  (analysis, 
attention,  relevance,  confidence,  and  satisfaction). 

9.  Four  variables — designer  background,  instructional  context,  designer  use  of  core  ISD  elements  and  designer  attitude 
toward  motivation — and  the  incorporation  of  motivation  and  its  components  (analysis,  attention,  relevance,  confidence, 
and  satisfaction). 

10.  The  incorporation  of  motivation  and  its  components,  (analysis,  attention,  relevance,  confidence,  and  satisfaction). 

11.  All  five  variables:  designer  background,  instructional  context,  designer  use  of  core  ISD  elements  and  designer  attitude 
toward  motivation,  and  the  incorporation  of  motivation  and  its  components  (analysis,  attention,  relevance,  confidence,  and 
satisfaction)  and  the  perceived  level  of  a project’s  success. 


Limitations  of  the  Study 

Some  recognized  limitations  are  involved  in  survey  studies.  First,  despite  a mailed  questionnaire’s  ability  to 
reach  large  numbers  of  people,  the  response  rate  is  often  low  (Knupfer  & McLellan,  1996),  which  may  make  it 
difficult  to  obtain  a representative  sample.  Second,  the  mailed  questionnaire  is  limited  to  volunteers,  who  may  be 
different  from  non-respondents  (Knupfer  & McLellan,  1996).  Third,  if  the  sample  is  drawn  from  members  of  a 
professional  organization,  the  sample  has  a possibility  of  being  skewed  toward  more  highly  trained  practitioners  than 
the  norm. 
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Figure  1.  Hypothesized  Model  of  Variables  Related  to  the  Incorporation  of  Motivational  Components  to 
Instructional  Design  and  the  Perceived  Level  of  the  Project's  Success 


Stepwise  multiple  linear  regression  analyses  and  path  analysis  techniques  were  used  to  answer  the  1 1 
research  questions  posed  for  this  study  and  test  the  hypothesized  model.  The  revised  model  was  developed  from  the 
results  of  stepwise  regression  analyses  and  path  analysis  techniques.  To  determine  the  relationships  between  the 
variable  sets,  regression  analyses  and  path  analysis  were  used.  The  results  of  these  analyses  are  depicted  in  the 
revised  model.  (See  Figure  2.) 

Discussion  of  Findings 

The  revised  model  depicts  the  variables  that  were  determined  to  be  valid  predictors  of  the  dependent 
variable  - perceived  success  of  the  project.  One  the  five  original  variable  sets,  designer  attitudes  toward  motivation, 
in  the  hypothesized  model  was  eliminated.  Four  of  the  variable  sets,  (a)  designer  background,  (b)  instructional 
context,  (c)  use  of  core  ISD  elements,  and  (d)  incorporation  of  motivation  design  components,  were  found  to  be 
valid  predictors.  Eight  variables  or  subcomponents  from  the  four  variable  sets  were  shown  to  have  a direct  effect  on 
the  dependent  variable,  perceived  level  of  the  project’s  success. 
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Figure  2.  Revised  Model  of  Variables  Related  to  the  Incorporation  of  Motivational  Components  to 
Instructional  Design  and  the  Perceived  Level  of  the  Project’s  Success 
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Designer  Background 

Respondents  rated  their  general  design  expertise  and  motivation  expertise.  Designer  expertise  was  not 
defined  in  the  questionnaire.  However,  designer  expertise  has  been  defined  in  the  literature  as  years  of  experience  in 
the  field  as  well  as  the  ability  to  internalize  and  apply  theoretical  principles  of  instructional  systems  design 
(Atchinson,  1996;  Rowland,  1992). 

Motivation  expertise  was  the  only  variable  from  designer  background  characteristics  that  directly  affected 
perceived  success  of  the  project.  This  finding  appeared  to  be  consistent  with  McCombs’  (1986)  conclusion  that 
expertise  in  design  is  one  of  the  critical  factors  leading  to  successful  application  of  an  ISD  model.  Finding  that 
motivation  expertise  has  a direct  effect  on  the  perceived  success  of  the  project  suggests  that  increased  experience 
with  motivational  design  increases  the  possibilities  for  a project’s  perceived  success.  Motivation  expertise  also  had 
an  indirect  effect  on  the  perceived  success  of  the  project  through  the  confidence  and  satisfaction  components  of 
motivational  design.  The  confidence  component  in  motivational  design  emphasizes  using  strategies  to  guild  a 
positive  expectation  for  success,  enhancing  the  student’s  beliefs  in  their  competence,  and  ensuring  personal  control 
of  learning  (Keller,  1987c).  The  satisfaction  component  in  motivational  design  emphasizes  helping  learners  feel 
positive  about  their  achievement. 

Confidence-building  strategies  may  be  applied  to  the  sequencing  of  content  such  as  ordering  problems  from 
easy  to  hard  to  help  ensure  success,  strategy  applied  successfully  to  computer  courseware  design  (Keller  & Suzuki, 
1988).  Additionally,  strategies  to  increase  confidence  may  be  used  to  influence  learners’  attitudes.  Motivational 
messages  were  used  effectively  to  increase  positive  attitudes  and  learner  confidence  regarding  the  completion  of  a 
difficult  course  conducted  in  a summer  workshop  for  teachers  in  Mozambique  (Visser  & Keller,  1990). 

The  influence  of  motivation  expertise  on  the  selection  of  confidence  and  satisfaction  strategies,  however, 
conflicts  with  Farmer’s  (1989)  research  that  showed  satisfaction  to  be  the  least  addressed  motivation  component  by 
designers.  One  explanation  for  this  difference  may  be  that  the  designers  in  Farmer’s  study  involved  novice  designers 
who  were  graduate  students  in  instructional  technology.  The  designers  in  the  present  study  had  a mean  of  13  years 
experience.  This  suggests  that  experienced  designers  with  practice  in  the  design  of  motivation  are  more  likely  to 
incorporate  confidence  and  satisfaction  strategies  from  the  ARCS  Model. 

These  findings  pertaining  to  confidence  and  satisfaction  are  not  consistent  with  research  by  Means, 
Jonassen,  and  Dwyer  (1997),  who  found  that  relevance  was  the  primary  component  that  influenced  motivation  and 
learning  outcomes.  Although  designers  in  the  present  study  cited  relevance  most  often  as  the  key  element  in 
motivation,  relevance  did  not  appear  as  a predictor  of  project  success.  These  contradictory  findings  about  motivation 
are  consistent  with  Farmer’s  (1989)  conclusions  about  designers’  decisions.  He  found  that  designers  were  more 
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likely  to  use  “their  personal  intuitive  feelings  about  what  was  motivating”  rather  than  follow  recommended  strategies 
suggested  by  the  ARCS  Model  (p.189).  Furthermore,  Farmer  noted  that  designers’  reluctance  to  incorporate 
motivation  into  instruction  systematically  stemmed  from  their  attitude  about  motivation.  He  observed  that  most 
subjects  in  his  study  viewed  motivation  as  “fuzzy  and  uncertain”  (p.  190). 

The  literature  is  inconclusive  about  the  effectiveness  of  applying  motivational  design  strategies  to  achieve 
desired  instructional  goals  (Means  et  al.,  1997).  Some  studies  have  found  positive  results,  while  others  have  found  no 
significant  effects  (Moller,  1994;  Naime-Deifenbach,  1991).  However,  one  explanation  for  varying  conclusions  may 
be  that  each  study  was  different  in  its  goals,  samples,  and  procedures.  Therefore,  more  research,  especially 
replication  studies,  regarding  the  uses  and  effectiveness  of  motivational  design  is  needed. 

Instructional  Context. 

Recent  research  into  the  role  of  context  suggests  that  contextual  variables  play  a role  in  successful  training 
and  learning  (Tessmer,  1995;  Tessmer  & Richey,  1997).  However,  there  is  little  information  about  the  possible 
relationship  between  where  the  designers’  products  are  used  (own  organization  or  client’s  organizations)  designers 
and  the  perceived  success  of  a project. 

Four  variables  from  the  instructional  context  variable  set  also  directly  affected  the  perceived  level  of  the 
project’s  success.  These  contextual  variables  were:  (a)  place  products  used:  another  organization,  (b)  classroom./ 
instructor- led  delivery,  (c)  amount  of  facilities  allocated,  and  (d)  importance  of  learners’  attitudes 

The  first  predictor  was  the  place  products  were  used.  The  respondents  indicated  where  their  products  of 
their  work  were  typically  used-in  their  own  organizations  or  in  a client’s  organization.  This  study  found  a positive 
relationship  between  use  of  the  designers’  instructional  products  in  the  clients’  organizations  and  perceived  success. 

It  was  not  clear  why  this  why  this  was  so.  One  possible  explanation  may  be  due  to  the  type  of  evaluations  general 
done  on  instructional  products.  Designers  and  trainers  commonly  use  “reaction,”  or  “smile  sheets,”  to  obtain  a first 
level  evaluation  of  the  instruction  (Kirkpatrick,  1994).  These  evaluations  are  often  constructed  by  the 
designer/trainer,  and  are  scored  and  interpreted  by  the  designer/trainer.  Therefore,  a limitation  of  this  study  is  that  the 
respondents  were  not  asked  s to  identify  the  types  of  evaluation  methods  they  used  to  reach  their  conclusion  about 
the  success  of  the  project.  Further,  there  were  no  objective  data  collected  from  the  learners  or  clients  on  the  success 
of  the  project.  The  determination  of  the  project’s  success  depended  entirely  on  the  respondents’  perceptions. 

Classroom/instructor  led  delivery  was  the  second  predictor  of  perceived  project  success.  The  positive 
relationship  between  the  two  variables  indicated  that  respondents  who  reported  greater  use  of  classroom  delivery 
tended  to  rate  the  project’s  success  higher.  Classroom  delivery/instructor  led  delivery  also  indirectly  influenced 
perceived  level  of  project  success  through  confidence  and  satisfaction.  Theses  positive  relationships  may  be  due  to 
the  fact  that  most  of  the  respondents  (n=166,  83%)  cited  instructor-led  classroom  instruction  as  the  delivery  type 
used  in  the  selected  project.  Although  respondents  could  have  indicated  more  than  one  type,  other  delivery  types 
such  as  print  (n=69,  34.5%),  computer  (n=63,  31.5$),  video  (n=26,  13%)  were  not  used  as  often  and  did  not  appear 
as  significant  predictors. 

The  third  contextual  variable  found  to  be  a significant  predictor  of  perceived  project  success  was  the 
amount  of  facilities  allocated.  However,  this  was  a negative  relationship,  which  suggested  that  participants  who 
estimated  the  amount  of  facilities  for  the  projects  as  low  or  moderate  were  more  likely  to  rate  the  success  of  the 
projects  higher.  This  finding  seems  to  contradict  research  by  Tessmer  and  Richey  (1997),  who  argue: 

The  physical  condition  of  the  immediate  environment  is  a potent  force  in  learning.  The  physical 
environment  does  not  so  much  increase  learning  when  it  is  excellent  as  inhibit  it  when  it  is  poor.  That  is,  a certain 
level  of  adequacy  must  be  attained  in  seating,  acoustics,  temperature,  and  lighting  for  proper  learning  to  take  place 
(P-97). 

However,  this  finding  may  not  be  completely  in  contrast  to  Tessmer  and  Richey’s  contextual  analysis. 
Respondents’  rated  the  “amount  of  facilities.”  as  low  or  moderate,  but  this  may  indicate  that  they  perceived  them  to 
be  “adequate.”  As  noted  previously,  more  than  80%  of  the  participants  in  this  study  indicated  that  their  project 
involved  classroom  instruction  with  approximately  one  third  using  video  and  or  computer  facilities.  The  “low”  to 
“moderate”  classroom  facilities  used  for  classroom  instruction  by  the  participants,  then,  was  deemed  sufficient  for 
the  perceived  level  of  project  success.  This  finding  may  indicate  that  the  amount  of  facilities  allocated  is  not  a 
critical  element  to  successful  instructor- led  classes  that  are  not  dependent  on  superior  facilities  and  equipment. 

Importance  of  learners’  attitudes  was  the  fourth  contextual  variable  to  directly  predict  the  perceived  level  of 
a project’s  success.  This  finding  indicates  that  respondents  who  reported  being  concerned  about  learners’  attitudes 
were  likely  to  perceive  the  project  as  being  successful.  This  finding  is  supported  by  the  literature.  Richey  (1992) 
notes  that  ISD  models  generally  address  learner  interests,  which  are  related  to  attitudes.  A leading  instructional 
design  textbook  (Rothwell  & Kazanas,  1992)  recommends  that  a needs  analysis  include  attitudes.  Since  Keller 
(1979)  and  others  (Bohlin,  Milheim,  & Viechnicki,  1991;  Bohlin  et  al.,  1990;  Keller,  1987b;  Keller,  1990;  Richey, 
1992;  Wlodkowski,  1985)  consider  learners’  attitudes  to  be  connected  to  motivation,  it  is  important  to  find  that 
practicing  designers  also  consider  attitudes  of  learners  to  be  important. 


Use  of  Core  ISD  Elements 

Respondents  rated  the  importance  core  ISD  elements  were  to  the  projects  identified  for  this  study.  The 
importance  of  using  an  ISD  model  (or  variable  “use  a model”)  was  found  to  be  a negative  predictor  of  the  perceived 
level  of  the  project’s  success.  This  finding  indicated  that  importance  of  using  an  ISD  model  does  not  have  a 
significant  relationship  to  the  perceived  success  of  the  project.  This  finding  was  not  consistent  with  the  theory  of 
ISD,  which  endorses  the  use  of  an  ISD  model  for  the  design  and  development  of  instructional  materials.  This  study’s 
finding,  however,  supports  research  on  designers  and  the  difference  between  designer  practice  and  theory  (Pieters  & 
Bergman,  1995;  Rowland,  1993;  Winer  & Vazquez-Abad,  1995).  This  finding  underscores  the  argument  by  Pieters 
and  Bergman,  who  note  that  . .design  is  often  tacit  (i.e.,  done  without  knowledge  that  a systematic  procedure  is 
being  followed)”  p.123. 

The  importance  of  using  an  instructional  design  model  (“use  a model”)  had  an  indirect  effect  through  the 
motivation  component,  confidence.  This  effect  was  positive,  indicating  that  instructional  designers  who  considered 
the  use  of  an  ISD  model  important  were  more  likely  to  include  confidence-building  strategies  into  the  design  of 
instructional  materials. 

Incorporation  of  Motivation  Design  Components. 

The  hypothesized  model  contained  five  subcomponents  in  the  variable  set,  Incorporation  of  Motivation 
Design  Components.  Based  on  the  ARCS  Model  (Keller,  1987c),  this  variable  set  included  the  following 
components:  (a)  motivation  analysis,  (b)  attention,  (c)  relevance,  (d)  confidence,  and  (e)  satisfaction.  According  to 
comments  made  by  the  respondents,  effective  instructional  design  includes  the  use  of  motivational  strategies. 
However,  the  revised  model  showed  that  only  confidence  and  satisfaction  were  directly  related  to  the  perceived  level 
of  the  project’s  success.  Both  of  these  variables  were  positive,  which  indicated  that  respondents  who  indicated  they 
emphasized  confidence  and  satisfaction  in  their  designs  were  likely  be  rate  the  success  of  the  project  as  high.  It  is  not 
clear  why  these  two  strategies  emerged  as  predictors  but  the  others  did  not.  An  explanation  may  be  in  the 
contradiction  that  exists  between  theory  or  beliefs  and  actual  practice.  The  majority  of  respondents  (121,  60%) 
agreed  with  the  statement  that  motivation  can  be  analyzed  effectively  and  applied  systematically  to  instructional 
design.  Nonetheless,  designers  in  this  study  appeared  less  likely  to  analyze  motivation,  but  seemed  likely  to  include 
one  or  more  of  motivation  components  (attention,  relevance,  confidence,  and  satisfaction)  when  planning, 
developing,  and  implementing  their  projects.  This  lack  of  motivational  analysis  supported  the  general  view  in  the 
field  that  motivation  is  somewhat  of  a vague  concept  that  is  difficult  to  be  analyzed  systematically  (Farmer,  1989; 
Keller,  1987c).  In  addition,  this  finding  showing  that  respondents  placed  more  importance  on  confidence  and 
satisfaction  components  than  attention  and  relevance  may  be  in  accordance  with  Keller’s  (Keller,  1987b) 
recommendation  for  the  selection  of  components  based  on  a learner  analysis  of  motivational  needs. 

Conclusions:  Implications  for  Practice  and  Recommendations  for  Research 

The  initial  problem  statement  presented  in  this  study  suggested  that  one  or  more  of  the  variables  in  the 
hypothesized  model  would  affect  a designer’s  application  of  motivational  components  to  the  design  process  and 
ultimately  a designer’s  perceived  success  of  the  project.  The  relationships  between  these  variables  were  analyzed. 

Because  this  study  surveyed  members  of  ISPI  who  are  involved  in  design  activities,  the  results  of  their 
observations  should  provide  several  implications  for  the  field. 


1 . Motivation  expertise  is  the  only  one  of  the  designer  background  characteristics  that  influences  the  designer’s  perceived 
success  of  the  project.  This  suggests  that  designer’s  should  make  an  effor  to  incorporate  motivational  design  strategies  to 
increase  the  likelihood  of  a successful  project. 

2.  Background  characteristics  such  as  age,  gender,  race/ethnicity,  and  education  level  do  not  appear  to  be  factors  that 
influence  the  designer’  incorporation  of  motivation  nor  the  designer’s  perceived  success  of  the  project.  While  these  factors 
may  be  important  considerations  for  a learner  analysis  regarding  the  design  of  motivational  strategies,  these  do  not  appear 
to  have  any  relationship  to  the  designer’s  incorporation  of  motivational  components  or  the  success  of  the  project. 

3.  Instructional  context  variables  regarding  the  importance  placed  on  learners’  attitudes  was  shown  to  have  a significant 
relationship  to  the  project.  Designers  should  make  sure  that  these  learner  characteristics  are  given  significant  consideration 
in  the  design  of  instructional  materials. 

4.  Classroom  or  instructor  led  delivery  is  another  contextual  variable  that  affects  the  perceived  success  of  a project.  This 
suggests  that  instruction  delivered  via  a classroom  with  an  instructor  has  increased  likelihood  for  success. 

5.  The  importance  of  using  of  an  ISD  model  in  the  design  of  instructional  materials  had  a negative  relationship  to  the 
perceived  success  of  the  project.  This  suggests  that  designers  probably  consider  the  reaction  to  the  end  product  as  more 
important  than  the  process  used  to  create  the  end  product.  Further,  this  may  indicate  that  experienced  designers  apply  tacit 
knowledge  to  the  design  process  and  incorporate  various  steps  from  different  models  rather  than  selecting  just  one  model. 


Recommendations  for  Further  Research 

The  findings  indicate  that  certain  designer  background  and  contextual  variables  have  significant 
relationships  to  the  incorporation  of  motivational  components  and  to  the  perceived  success  of  a project.  The  data 
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show  motivation  expertise  is  an  important  factor  in  predicting  the  success  of  a project  and  that  certain  contextual 
factors  and  design  practice  variables  influences  the  outcome  of  a project.  However,  there  are  several  issues  that  need 
further  research. 

Expertise  had  a positive  impact  on  the  perceived  success  of  a project  and  the  incorporation  of  motivation 
analysis.  However,  this  study  did  not  define  or  describe  motivation  expertise.  Rowland  studied  instructional  design 
expertise  but  motivational  design  expertise  has  not  been  a subject  of  research.  More  needs  to  be  done  to  describe  and 
define  the  practice  of  motivational  design. 

The  two  motivational  components  found  to  have  the  most  impact  on  perceived  success  of  the  project  are 
confidence  and  satisfaction.  A study  by  Means  and  his  colleagues  show  relevance  is  the  most  important  motivational 
component.  Keller,  however,  asserts  that  all  four  components  are  necessary,  but  one  or  two  may  be  emphasized  more 
based  on  a thorough  learner  and  content  analysis.  Further  research  needs  to  be  done  to  determine  if  one  component  is 
more  essential  to  motivation  than  the  others. 

Most  of  the  designers  in  this  study  designed  instruction  for  classroom  delivery.  Since  a greater  emphasis  is 
being  placed  on  designing  for  technology,  it  would  be  interesting  to  replicate  this  study  with  a sample  population  of 
designers  who  design  primarily  for  computer  or  web-based  delivery. 

Most  of  the  practitioners  said  that  they  received  little  formal  training  in  design  of  motivational  strategies. 
Further  research  is  needed  into  the  university  programs  to  determine  whether  or  not  instructors  are  including 
motivational  design  in  their  course  content.  Further,  it  would  be  helpful  to  try  to  determine  the  factors  that  influence 
their  decisions. 

This  study  found  that  the  use  of  an  ISD  model  had  a negative  influence  on  the  perceived  success  of  the 
project.  This  conclusion  was  consistent  with  studies  about  the  gap  between  theory  and  design  practice,  but  it  is 
inconsistent  with  ISD  theory.  More  research  needs  to  be  done  on  how  designers  make  decisions  and  how  they  use 
ISD  models  as  well  as  the  factors  that  influence  their  choices. 

Finally,  this  survey  study  depended  on  self  report  rather  than  on  direct  observation  of  designers.  It  would  be 
interesting  to  study  the  actual  practice  of  designers  and  their  products  to  determine  the  motivational  components  they 
incorporate  into  the  design  of  instructional  materials  and  the  reason  for  their  decisions.  It  is  important  for  the  field  to 
continue  to  study  the  gaps  between  theory  and  practice  and  the  possible  reasons  for  the  gaps.  And  it  is  important  to 
try  to  determine  the  factors  that  lead  to  a project’s  success. 
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The  purpose  of  this  study  was  to  examine  the  effects  of  a prototype  of  motivationally -adaptive  CAI, 
developed  in  accordance  with  the  ARCS  model  of  motivational  design,  on  achievement,  perceived 
motivation  (both  overall  motivation  and  each  of  A,  R,  C,  & S components),  efficiency,  and  continuing 
motivation.  Students  studied  one  of  three  versions  of  CAI:  motivationally -adaptive  CAI,  motivationally- 
saturated  CAI,  or  motivationally-minimized  CAI.  Motivationally-adaptive  CAI  showed  higher  effectiveness, 
overall  motivation,  and  attention  than  the  other  two  CAIs.  For  relevance,  motivationally-adaptive  CAI  was 
higher  than  motivationally-saturated  CAI  but  not  higher  than  motivationally-minimized  CAI.  For 
confidence  and  satisfaction,  motivationally-  adaptive  CAI  was  not  higher  than  the  other  two  CAIs.  For 
efficiency,  both  motivationally-adaptive  CAI  and  motivationally-minimized  CAI  were  higher  than 
motivationally-saturated  CAI,  but  only  the  efficiency  of  the  motivationally-adaptive  CAI  was  identified  as 
having  practical  importance.  For  continuing  motivation,  there  were  no  significant  differences  among  the 
three  CAIs,  but  a significant  correlation  was  found  between  overall  motivation  and  continuing  motivation 
across  three  CAIs.  These  findings,  first,  support  the  contention  that  the  ARCS  model  is  useful  for 
developing  motivationally-  adaptive  CAI.  Second,  motivationally-adaptive  CAI  can  be  an  effective, 
efficient,  and  motivating  form  of  instruction  which  may  also  enhance  learners ’ continuing  motivation. 
Related  discussions  and  recommendations  for  future  study  are  provided. 

Computer  mediated-instruction  is  being  widely  used  as  more  advanced  computer  and  communication 
technology  becomes  available.  Learners,  either  school-aged  or  adult,  now  learn  and  will  have  to  learn  more  from 
computer- mediated  instruction  such  as  computer-assisted  instruction  (CAI),  intelligent  tutoring  systems,  integrated 
learning  systems,  or  network-based  learning  environments.  This  trend  shows  that  computers  as  instructional  media 
are  becoming  now  a routine  part  of  instructional  delivery  systems. 

However,  two  recent  concerns  regarding  the  wide  spread  used  of  computers  have  indicated  the  need  to  deal 
with  motivation  to  learn  during  the  computer- mediated  instruction.  One  concern  is  that  as  computers  are  becoming 
widely  used  for  instructional  delivery,  their  novelty  effects  will  disappear  (Keller  & Suzuki,  1988;  Keller,  1997). 
Students  will  no  longer  be  fascinated  with  learning  from  computer-assisted  instruction  as  much  as  they  once  were. 
Then  the  question  is  how  to  get  hold  of  their  motivation  during  the  computer-mediated  instruction. 

Several  advocates  of  adaptive  CAI  have  raised  the  other  concern,  which  is  how  to  include  adaptive 
responses  to  student  motivation.  They.have  pointed  out  that  few  previous  efforts  to  design  adaptive  CAI  have 
considered  learner  motivation  as  a factor  for  which  subsequent  instruction  is  prescribed  (del  Soldato  & du 
Boulay,1995;  Hativa  & Lesgold,  1991;  McCombs,  Eschenbrenner,  Jr.,  & O’Neil,  Jr.,  1973).  Their  main  concern  has 
been  that  although  possible  benefits  of  developing  adaptive  CAI  have  been  found,  one  limitation  of  prior  efforts 
(e.g.,  Atkinson,  1976;  Holland,  1977;  Houlihan,  Finkelstein,  & Johnson,  1992;  Mills  & Ragan,  1994;  Ross  & 
Morrison,  1988;  Tennyson  & Christen,  1988;  Tennyson  & Park,  1980)  is  that  the  students’  motivation  to  learn  is 
disregarded  or  assumed  to  be  embedded  in  the  cognitively  adaptive  CAI.  Therefore,  they  have  proposed  the 
development  of  a motivationally-adaptive  CAI  that  is  adjusted  to  motivational  changes  for  better  learning  outcomes, 
i.e.,  increased  achievement  and  positive  attitude  toward  learning. 

Three  approaches  to  the  design  and  development  of  a motivating  CAI  can  be  found  in  the  literature.  The 
first  is  the  Computer  Feature  Approach,  in  which  specific  computer  features  and  novelty  effects  (Clark,  1983)  are 
assumed  to  increase  the  appeal  of  CAI.  For  example.  Brown  pointed  out,  “the  student  is  rewarded  by  the  use  of  the 
machine  itself  ” (Brown,  1986,  p.  28).  Also,  it  has  been  contended  that  “the  cumulative  effects  of  entities  such  as 
color,  graphics,  and  animation  can  be  instructionally  and  motivationally  powerful”  (Relan,  1992,  p.  619).  The  second 
is  the  Principle  Seeking  Approach,  in  which  prescriptive  motivational  design  principles  and  tactics  for  CAI  are 
identified  or  developed  diverse  theoretical  and  practical  perspectives.  For  example,  instructional  games  are 
recommended  for  the  development  of  motivating  CAI  (Dempsey,  Lucassen,  Gilley,  & Rasmussen,  1993;  Malone, 
1981;  Malouf,  1988).  Cooperative  learning  on  CAI  (Johnson  & Johnson,  1986)  is  also  suggested,  as  is  learner 
control  of  CAI  (Klein  & Keller,  1990).  The  third  is  the  Model  Establishing  Approach.  The  relationships  between 
motivational  theories  and  computer  features  are  identified  and  incorporated  into  a practical  model  for  designing 
motivating  CAI.  For  example,  Lee  & Boling  (1996)  proposed  the  framework  of  motivational  screen  design.  Keller 
and  Suzuki  (1988)  proposed  a set  of  motivational  strategies  for  designing  a motivating  courseware  based  on  the 
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ARCS  model.  Keller  & Keller  (1991)  and  Keller  & Keller  (1992)  later  developed  this  into  a framework  for 
designing  motivating  multimedia  instruction. 

All  of  these  three  approaches  may  be  used  for  the  development  of  a motivating  CAL  However,  they  all  have 
limitations.  The  Computer  Feature  Approach  does  not  address  the  decreasing  motivational  effects  of  computer 
features  whose  novelty  effects  will  disappear  as  learners  become  more  familiar  with  computers.  The  Principle 
Seeking  Approach  has  a value  in  that  it  provides  more  specific  and  practical  guidelines.  However,  it  is  limited  in  that 
its  approach  tends  to  focus  on  limited  and  specific  aspects  of  motivation,  whereas  motivation  to  learn  needs  to  be 
understood  more  comprehensively.  For  example,  not  all  CAI  programs  can  be  developed  as  instructional  games.  The 
Model  Establishing  Approach  appears  to  be  the  most  useful  in  that  it  provides  a frame  of  reference  that  will  guide 
instructional  designers.  But  it  is  also  incomplete  in  that  it  suggests  that  all  the  motivational  strategies  need  to  be 
identified  and  pre-sequenced  before  instructional  delivery. 

Furthermore,  there  is  a common  problem  across  the  three  approaches.  That  is,  they  do  not  address  the 
continuously  changing  nature  of  motivation  to  learn.  One  of  the  reasons  for  this  could  be  that  all  of  the  decisions 
about  what  types  of  motivational  tactics  to  include  have  to  be  made  at  the  design  stage  and  cannot  be  modified  once 
the  instruction  is  ready  for  delivery.  This  means  that  every  student  receives  exactly  the  same  package.  However, 
students  who  are  highly  motivated  before  starting  CAI  will  not  always  remain  motivated  throughout  the  whole 
learning  process.  And  conversely,  some  students  who  are  not  motivated  before  beginning  CAI  may  become 
motivated  as  they  proceed  through  the  process.  Therefore,  it  would  be  highly  desirable  to  enable  the  CAI  package  to 
be  responsive  to  motivational  differences  within  the  same  learner  at  different  points  in  time. 

Regarding  these  problems  of  the  three  traditional  approaches,  a recommendation  has  been  made 
consistently  in  the  literature  of  motivation  that  we  should  provide  optimal  motivational  strategies  to  learners 
(Brophy,  1987;  Lepper,1988).  If  learners  are  already  motivated,  they  should  not  be  exposed  to  any  more  unnecessary 
motivational  tactics  (Astleitner  & Keller,  1995;  Keller,  1983,  1987  a,b,c).  In  that  case,  just  sustaining  their 
motivation  is  recommended.  In  contrast,  learners’  motivation  should  be  enhanced  when  they  are  de-motivated.  In  this 
study,  as  a way  of  providing  optimal  motivational  strategies,  the  idea  of  adaptive  provision  of  motivational  strategies 
was  tested  on  computer-assisted  instruction. 

A few  pioneering  efforts  have  discussed  and  tried  the  development  of  motivationally  adaptive  CAI 
(Astleitner  & Keller,  1995;  del  Soldato  and  du  Boulay,1995;  Rezabek,  1994).  However,  these  efforts  have  several 
shortcomings  First,  few  have  been  supported  by  positive  empirical  data  with  respect  to  increased  performance  and 
motivation.  For  example,  Rezabek  just  discussed  the  use  of  intrinsic  motivational  strategies  for  the  development  of 
motivationally-adaptive  instructional  system.  Astleitner  & Keller  (1995)  suggested  a simulation  approach  for 
designing  motivationally-adaptive  CAI,  but  they  mainly  provided  ideas  on  how  a computer  can  predict  motivational 
states.  Reporting  the  positive  formative  evaluation  results  on  the  program  called  “MORE”,  del  Soldato  and  du 
Boulay  (1995)  suggested  the  need  for  further  study  on  its  effects  on  performance  and  motivation.  Therefore,  studies 
to  investigate  the  actual  effects  of  motivationally-adaptive  CAI  on  outcomes  such  as  achievement,  perceived 
motivation,  continuing  motivation,  and  efficiency  need  to  be  conducted. 

Second,  the  content  of  instruction  has  been  mostly  composed  of  a battery  of  test-items.  For  example,  del 
Soldato  and  du  Boulay  (1995)  developed  an  intelligent  tutoring  system  where  the  system  presented  test-items  to  be 
solved  by  learners,  and  then  both  instructional  strategies  and  motivational  strategies  were  prescribed  based  on 
student  performance.  However,  there  is  no  reason  that  contents  in  the  motivationally-adaptive  CAI  should  be  limited 
to  test-items.  Therefore,  there  needs  to  be  a study  that  investigates  the  possibility  of  developing  motivationally- 
adaptive  CAI  which  teaches  diverse  types  of  content  such  as  verbal  information,  concepts,  principles,  etc.  (Gagne, 
1985).  Regarding  this  need,  del  Soldato  & du  Boulay  suggested  that  “a  richer  domain  representation,  including,  for 
example,  a wider  variety  of  links  between  topics,  would  provide  space  for  further  elaboration  of  motivational 
tactics.”  (p.  373) 

Third,  the  previous  ideas  suggested  by  Astleitner  & Keller  (1995),  and  del  Soldato  and  du  Boulay  (1995) 
require  challenging  efforts  for  instructional  designers  to  understand  and  utilize  and  utilize  their  models  for  designing 
instruction.  For  example,  Astleitner  & Keller’s  approach  requires  instructional  designers  to  obtain  parameters,  such 
as  incentive  value,  to  be  used  for  simulation.  Del  Soldato  and  du  Boulay’s  approach  may  require  instructional 
designers  to  face  the  complexity  of  programming  intelligent-tutoring  systems.  Therefore,  it  would  be  desirable  to 
produce  a generic,  motivationally-adaptive  CAI  which  can  be  designed  in  a simpler  and  more  generalizable  way. 

This  product  would  provide  confidence  to  instructional  designers  in  dealing  with  motivational  problems  in  designing 
CAI  because  they  would  not  need  to  worry  about  all  the  complicated  computer-related  logic  and  calculations. 

Finally,  there  is  a need  to  use  an  integrated  model  which  will  guide  instructional  designers.  Park  (1996) 
points  out  that  a micro-adaptive  instructional  system  using  computer  technology  has  been  based  on  “a  particular 
model  or  theory  of  learning,  and  its  adaptation  of  the  learning  environment  is  rather  sophisticated  (p.  646).” 

Likewise,  also  for  the  development  of  motivationally-adaptive  CAI,  we  need  to  find  a practical  model  that  is 
theoretically  validated  and  practically  field-tested.  In  fact,  one  reason  that  there  has  not  been  much  effort  made  to 
design  a motivationally-adaptive  CAI  may  be  the  lack  of  a practical  model  which  can  guide  the  diagnosis  and 
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prescription  (Astleitner  & Keller,  1995).  Although  McCombs,  Eschenbrenner,  Jr.,  & O’Neil,  Jr.(1973)  drew 
attention  to  the  need  for  designing  the  motivationally-adaptive  CAI  in  1973,  it  was  not  until  recently  that  actual 
efforts  were  made  to  develop  such  a system. 

In  this  study,  the  ARCS  model  is  used  as  a framework  for  the  development  of  motivationally-adaptive  CAI. 
Motivation  is  something  inside  the  learner's  mind,  and  it  is  important  to  stimulate  or  sustain  learners'  motivation  to 
leam  for  their  active  learning.  As  a way  of  addressing  this  issue,  the  ARCS  (Attention,  Relevance,  Confidence,  and 
Satisfaction)  model  was  proposed  by  Keller(1979, 1983, 1987a, b,c).  He  developed  it  as  a conceptual  tool  and  process 
for  designing  instruction  that  is  appealing,  in  addition  to  being  efficient  and  instructionally  effective. 

Generally,  in  the  case  of  external  adaptation,  an  adaptive  CAI  needs  to  have  two  functions  which  are 
essential  to  meet  instructional  needs  (Ross,  Rakow,  and  Bush,  1980;  Carrier  & Jonassen,  1988).  First,  an  adaptive 
CAI  needs  to  diagnose  the  state  of  learners.  That  is,  learner  characteristics  to  which  instruction  is  adapted  must  be 
continuously  identified  and  measured.  Second,  the  adaptive  CAI  should  have  the  capability  to  provide  the  most 
appropriate  instruction  to  address  the  identified  state  of  learners.  Similarly,  motivationally-adaptive  CAI  should  have 
the  same  two  characteristics. 

For  example,  CAI  cannot  be  motivationally  adaptive  if  it  fails  to  identify  the  learner’s  motivational  levels  at 
more  than  one  point  of  time  in  learning.  Because  the  learner’s  motivational  level  can  continuously  change  throughout 
instruction,  any  CAI  which  is  developed  based  on  a one-time  motivational  analysis  prior  to  developing  a course  may 
not  work.  Also,  motivationally-adaptive  CAI  should  be  able  to  provide  instruction  which  is  precisely  adapted  to 
meet  the  learner’s  motivational  state. 

This  need  requires  us  to  use  the  ARCS  model  differently  from  the  way  it  has  been  used  for  motivational 
design.  In  fact,  the  ARCS  model  has  been  applied  to  the  enhancement  of  different  types  of  instruction  with  respect  to 
content  and  method  of  delivery.  For  example,  with  respect  to  delivery,  it  has  been  applied  to  classroom  instruction, 
self-directed  learning  materials,  and  mediated  instruction  such  as  CAI  (Keller  & Keller,  1991;  Keller  & Suzuki, 

1988).  Its  primary  application  has  been  in  the  initial  development  of  materials  based  on  the  entry-level  motivation  of 
students 

With  this  in  view,  the  central  question  of  this  study  is  to  investigate  whether  the  ARCS  model  can  help 
instructional  designers  design  CAI  which  is  motivationally  individualized  or  adaptive  to  the  learners'  motivational 
needs.  Keller's  ARCS  model  seems  to  provide  a basis  for  meeting  the  above  two  functions,  i.e.,  diagnosis  and 
prescription,  of  a motivationally  adaptive  CAI.  For  the  former,  audience  analysis  is  emphasized,  and  for  the  latter, 
Keller  & Suzuki  (1988)  propose  sets  of  prescriptive  strategies  based  on  the  ARCS  model. 

Moreover,  the  ARCS  model  is  developed  from  a thorough  literature  review  on  motivational  theory.  The 
four  components,  attention,  relevance,  confidence,  and  satisfaction,  are  based  on  a general  macro  theory  of 
motivation  (Keller,  1979,1983)  and  have  been  verified  through  several  validation  studies  and  discussions  (Bickford, 
1989;  Keller,  1984;  Means,  Jonassen,  & Dwyer,  1997;  Klein  & Freitag,1992;  Newby,  1991;  Nwagbara,  1993;  Small 
& Gluck,  1994;  Visser  & Keller,  1990).  Mostly,  the  use  of  the  ARCS  model  has  been  supported  by  empirical  results 
showing  increased  motivation,  achievement,  and  continuing  motivation.  However,  no  study  has  been  conducted  to 
investigate  the  use  of  the  ARCS  model  for  the  development  of  the  motivationally-adaptive  CAI.  To  apply  it,  there 
are  two  major  steps:  diagnosis  of  motivational  states  and  prescription  of  tactics. 

Diagnosing  Motivational  State 

Keller  (1983,1987  a,  b,  c)  emphasized  the  importance  of  audience  motivational  analysis  before  designing 
motivational  enhancements  to  instruction.  The  learner’s  motivational  level  can  be  estimated  based  on  the  designer’s 
personal  experience,  objectively  assessed  from  collected  and  analyzed  data,  or  by  both  (1987b).  An  audience 
motivational  analysis  is  generally  conducted  for  each  component  of  attention,  relevance,  confidence,  and  satisfaction, 
although  the  decisions  as  to  how  specific  to  be  will  depend  on  the  criticality  of  the  decision,  the  anticipated 
obstacles,  and  the  consequences  of  failure. 

For  example,  for  an  instructor- led  course,  the  process  questions  (Figure  1)  presented  by  Keller  (1987a  or  b) 
can  be  used  as  checklists  because  they  are  well  categorized  into  four  components  of  motivation  (ARCS),  enabling 
comprehensive  check-ups.  Designers  can  use  those  questions  for  their  “best  guess”  process  or  for  developing  an 
instrument  for  measuring  motivation.  In  either  case,  it  is  the  designers’  initial  responsibility  to  identify  the  learners’ 
levels  of  motivation.  An  instructor  can  then  more  or  less  continuously  monitor  motivation  and  change  tactics  as 
necessary.  This  fact  has  an  important  implication  for  instructional  designers  when  they  try  to  use  the  ARCS  model 
for  designing  the  motivationally-adaptive  CAI:  there  should  be  measurement  tools  embedded  in  the  instructional 
program  which  continuously  monitor  learners’  motivational  change  as  an  experienced  tutor  does.  For  example,  the 
audience  analysis  needs  to  be  conducted  several  times  throughout  the  instruction. 
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Figure  1 Process  Questions  and  Motivational  Strategies  ( adapted  from  Keller ; 1987a) 


Questions  for  Components 

Suggested  Strategies 

Attention 

• What  can  I do  to  capture  their  interest? 

• How  can  I stimulate  an  attitude  of  inquiry? 

• How  can  I maintain  their  attention? 

• Create  curiosity,  wonderment  by  using  novel  approaches, 
injecting  personal  and/or  emotional  material. 

• Increase  curiosity  by  asking  questions,  creating  paradoxes, 
generating  inquiry,  and  nurturing  thinking  challenges. 

• Sustain  interest  by  variations  in  presentation  style,  concrete 
analogies,  human-interest  examples,  and  unexpected  events. 

Relevance 

• How  can  I best  meet  my  learner’s  needs? 
(Do  I know  their  needs?) 

• How  and  when  can  I provide  my  learners 
with  appropriate  choices,  responsibilities, 
and  influences? 

• How  can  I tie  the  instruction  to  the 
learner’s  experiences? 

• Provide  statements  or  examples  of  the  utility  of  the  instruction 
and  either  present  goals  or  have  learners  define  them. 

• Make  instruction  responsive  to  learner  motives  and  values  by 
providing  personal  achievement  opportunities,  cooperative 
activities,  leadership  responsibilities,  and  positive  role  models. 

• Make  the  material  and  concepts  familiar  by  providing  concrete 
examples  and  analogies  related  to  the  learner's  work. 

Confidence 

• How  can  I assist  in  building  a positive 
expectation  for  success? 

• How  will  the  learning  experience  support 
or  enhance  the  students’  beliefs  in  their 
competence? 

• How  will  the  learners  clearly  know  their 
success  is  based  on  their  efforts  and 
abilities  . 

• Establish  trust  and  positive  expectations  by  explaining  the 
requirements  for  success  and  the  evaluative  criteria. 

• Increase  belief  in  competence  by  providing  many,  varied,  and 
challenging  experiences  which  increase  learning  success. 

• Use  techniques  that  offer  personal  control  (whenever  possible) 
and  provide  feedback  that  attributes  success  to  personal  effort. 

Satisfaction 

• How  can  I provide  meaningful 
opportunities  for  learners  to  use  their 
newly-acquired  knowledge  and  skill? 

• What  will  provide  reinforcement  to  the 
learners’  successes? 

• How  can  I assist  the  student  anchoring  a 
positive  feeling  about  their 
accomplishments? 

• Provide  feedback  and  other  information  that  reinforces 
positive  feelings  for  personal  effort  and  accomplishment. 

• Use  verbal  praise,  real  or  symbolic  rewards,  and  incentives,  or 
let  learners  present  the  results  of  their  efforts  ("show  and  tell") 
to  reward  success. 

• Make  performance  requirements  consistent  with  stated 
expectations,  and  provide  consistent  measurement  standards 
for  all  learners'  tasks  and  accomplishments. 

Two  issues  arise  when  audience  analysis  is  conducted  by  the  computer.  One  is  the  need  for  comprehensive 
assessment  of  motivation,  the  other  is  the  method  of  assessment.  As  far  as  comprehensiveness  is  concerned,  although 
motivational  levels  of  all  four  components  (attention,  relevance,  confidence,  and  satisfaction)  can  be  assessed  in  the 
case  of  the  instructor-led  course,  CAI  has  some  limitations  in  this  regard. 

The  reason  for  measuring  motivation  in  adaptive  CAI  is  to  provide  formative  input  that  can  be  used  to 
determine  what  motivational  tactics  to  use  in  the  subsequent  part  of  the  program.  The  measures  of  attention, 
relevance,  and  confidence  can  easily  be  used  for  formative  feedback.  Measures  of  satisfaction  that  are  taken  at 
intervals  during  the  program  could  also  be  used  for  formative  feedback  and  for  the  selection  of  subsequent 
motivational  tactics,  but  in  many  respects  satisfaction  measures  will  be  more  summative  in  nature.  This  will  be 
especially  true  in  relatively  short  episodes  of  instruction.  Although  the  analysis  of  satisfaction  can  be  conducted 
several  times,  it  is  difficult  to  provide  adaptive  motivational  strategies  to  address  the  measured  level  of  satisfaction. 
Satisfaction  could  be  eliminated  from  this  initial  study  of  adaptive  CAI  without  affecting  the  theoretical  or  practical 
foundation  of  the  study.  It  is  important  however  to  use  all  three  of  the  other  categories  because  of  their  interactive 
nature.  If  the  treatment  were  long,  with  several  lessons  or  units  of  work,  then  it  would  be  important  to  monitor 
satisfaction  of  each  point. 

Second,  the  assessment  method  to  be  used  in  audience  analysis  is  another  concern.  Although  Keller  (1987b) 
contends  that  the  full  range  of  measurement  possibilities  can  be  considered,  in  the  case  of  CAI  there  is  an  inevitable 
limitation  that  it  is  the  computer,  not  a human  tutor,  who  assesses  or  monitors  the  learner’s  motivational  change. 

In  this  study,  self-report  measures  were  used  for  embedded  audience  analyses  throughout  the  instruction. 
Although  self-report  measures  can  have  problems  with  respect  to  reliability  and  validity,  the  researcher  judged  that  it 
might  be  the  simplest  method  which  can  be  easily  understood  and  utilized  by  most  instructional  designers.  Keller 
(1987b)  says  that  straightforward  self-report  measures  can  be  very  useful  when  they  focus  on  an  identified  area  of 
concern.  For  example,  in  this  study,  each  learner  was  asked  to  report  his  or  her  motivation  in  terms  of  its  direction 
and  magnitude  with  respect  to  attention,  relevance,  and  confidence.  In  addition,  as  a way  of  complementing  the 
weakness  of  self-report  measures,  measures  of  performance  scores  on  the  quiz  were  also  used  in  conjunction  with 
their  motivational  self-assessments. 
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Prescribing  Motivational  Strategies 

Based  on  the  ARCS  model,  Keller  and  Suzuki  (1988)  suggested  a set  of  motivational  strategies  for 
designing  a CAI  program.  They  say  that  CAI  developed  with  the  ARCS  model  is  expected  to  be  motivating 
compared  to  ones  not  developed  in  accordance  with  the  ARCS  model.  Of  course,  any  CAI  can  be  motivating 
provided  that  the  instructional  designer’s  skills,  experience,  and  creativity  are  adequate  whether  or  not  they  have  any 
knowledge  of  the  ARCS  model.  The  ARCS  model  simply  tries  to  codify  this  knowledge  gained  from  the  research 
literature  and  the  examples  of  successful  practitioners.  However,  there  can  be  a benefit  to  using  the  ARCS  model,  or 
some  other  systematic  approach  to  motivational  design  for  guidance  and  for  documenting  results  in  a way  that  can  be 
used  by  others.  This  is  where  Keller’s  ARCS  model  and  motivational  strategies  suggested  by  Keller  and  Suzuki 
could  be  used  with  its  systematic  but  holistic  problem  solving  approach.  Motivational  strategies,  well  organized  in 
terms  of  ARCS  components,  are  expected  to  help  ensure  the  motivational  quality  of  CAI. 

However,  little  research  has  been  done  on  whether  the  adaptive  use  of  motivational  strategies  suggested  by 
Keller  & Suzuki  can  increase  the  appeal  and  effectiveness  of  CAI  by  being  adjusted  to  motivational  changes  of 
learners,  compared  with  those  used  in  a CAI  program  which  is  not  motivationally-adaptive.  Visser  and  Keller  (1990) 
applied  the  ARCS  model  in  an  adaptive  way  and  reported  promising  results  for  an  ins  true  tor- led  course.  Their 
application  was  “based  on  continuous  diagnosis  of  the  audience  and  subsequent  formulation  or  reformulation  of  a 
solution  (p.  472).” 

Building  on  this  concept  and  other  research,  the  purpose  of  this  study  was  to  investigate  the  use  of  the 
ARCS  model  in  developing  a motivationally-adaptive  CAI  which  is  appealing,  will  enable  learners  to  study  more 
effectively  and  efficiently,  and  will  enable  learners  to  have  more  continuing  motivation.  To  achieve  this  purpose,  it 
was  necessary  to  prepare  a prototypical  CAI  that  is  motivationally-adaptive  and  to  compare  it  to  two  different 
motivationally  non-adaptive  versions  of  CAI.  The  first  non-adaptive  version  (motivationally-saturated  CAI)  is 
enhanced  with  a full  set  of  motivational  strategies,  and  the  second  (motivationally-minimized  CAI)  has  virtually  no 
motivational  strategies  added. 

It  was  expected  that  the  motivationally-adaptive  CAI  would  be  most  motivating  to  learners  because  it  was 
designed  to  a)  avoid  providing  excessive  motivational  strategies  that  would  distract  or  annoy  learners  who  already 
found  the  instruction  to  be  motivating,  and  b)  to  overcome  motivational  deficiencies  among  learners  who  were  bored 
or  otherwise  demotivated  by  the  content.  It  was  also  expected  that  the  motivationally-adaptive  CAI  would  be  most 
effective  in  terms  of  learning  achievement,  and  successful  in  producing  the  highest  continuing  motivation.  Finally, 
the  efficiency  of  the  three  versions  of  CAI  s was  determined  by  comparing  achievement  to  time  spent  on  the  lesson. 

It  was  expect  that  the  motivationally-adaptive  version  would-be  most  efficient. 

Method 

Subjects  and  Design 

Subjects  were  66  tenth  grade  students  from  the  Developmental  Research  School  (DRS)  affiliated  with  a 
large  university  in  the  Southeastern  United  States.  Students  are  selected  and  admitted  to  the  DRS  to  be  representative 
of  Florida’s  school-age  population  in  terms  of  academic  ability,  race,  sex,  and  socio-economic  status.  Subjects  were 
selected  from  three  different  classes  taught  by  the  same  biology  teacher.  The  classes  represented  two  different  levels 
of  ability,  with  the  higher  being  the  college  prep  group. 

Subjects  who  had  been  identified  by  the  biology  teacher  as  not  having  studied  the  contents  were  randomly 
assigned  to  three  conditions  of  one  independent  variable  (motivationally-adaptive  CAI,  motivationally-saturated 
CAI,  and  motivationally-minimized  CAI).  The  dependent  variables  were  effectiveness  (learning),  efficiency  (amount 
of  learning  per  time  spent),  motivation  (both  overall  motivation  and  each  of  attention,  relevance,  confidence,  and 
satisfaction),  and  continuing  motivation. 


Materials 

Three  versions  of  hypercard  computer-assisted  instruction  were  developed  that  contained  the  same  contents 
on  Genetics.  The  contents  were  adopted  and  modified  from  three  sources:  a print-based  biology  instructional  module 
developed  by  Osman  (1992),  a hypertext  CAI  developed  by  Park  (1993),  and  a textbook  entitled  Biology  (Goodman 
et  al.,  1989).  Genetics  was  chosen  for  three  reasons.  First,  the  content  of  the  instruction  needed  to  be  facts,  concepts, 
and  principles  (Gagne,  1985)  as  mentioned  earlier  in  this  paper.  Second,  the  reading  ability  level  was  appropriate  to 
tenth  grade  students  (Park,  1993).  Third,  it  was  judged  that  the  content  needed  to  be  somewhat  technical  and  must 
not  be  easily  understood  without  student  effort.  It  was  confirmed  through  interviews  with  biology  teachers  that, 
although  the  students  appeared  to  know  much  about  genetics,  in  fact,  their  understanding  of  the  details  tended  to  be 
poor. 

Each  version  of  CAI  included  three  sections:  first,  (Introduction,  Gregor  Mendel,  Mendel’s  Experiment, 
The  Principle  of  Dominance);  second,  (The  Principle  of  Segregation,  The  Principle  of  Independence  Assortment, 
The  Principle  of  Chance  and  Probability),  and  third,  (Predicting  the  Results  of  Genetic  Crosses,  The  Punnett 
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Square).  Students  using  the  three  versions  of  CAI  were  asked  by  the  computer  for  their  motivational  attitudes  just 
prior  to  beginning  each  of  the  three  sections.  Each  time,  they  were  asked  about  their  attitudes  toward  the 
interestingness  (attention),  usefulness  (relevance),  and  confidence  in  learning  the  instruction.  Immediately  following 
the  motivational  analysis,  they  were  presented  with  their  assigned  version  of  the  instructional  material 
(motivationally-saturated,  adaptive,  or  minimized).  At  the  end  of  the  instruction,  they  were  asked  the  continuing 
motivation  question.  The  flow  of  events  is  illustrated  in  Figure  2 

Figure  2 Flow  of  instruction  in  three  versions  of  CAI 


1 . Present  the  introduction  and  first  embedded  motivational  analysis 

I find  the  subject  of  genetics  interesting.  Yes  or  No 

Learning  genetics  will  be  useful  to  me.  Yes  or  No 

I feel  confident  that  I can  leam  genetics.  Yes  or  No 

2.  Present  the  first  section  of  instruction  and  four-item  quiz 

3.  Present  the  second  embedded  motivational  analysis 

I find  the  subject  of  genetics  interesting.  Yes  or  No 

Learning  genetics  will  be  useful  to  me.  Yes  or  No 

I feel  confident  that  I can  leam  genetics.  Yes  or  No 

4.  Present  the  second  section  of  instruction  and  four-item  item  quiz 

5.  Present  the  third  embedded  motivational  analysis 

I find  the  subject  of  genetics  interesting.  Yes  or  No 

Learning  genetics  will  be  useful  to  me.  Yes  or  No 

I feel  confident  that  I can  leam  genetics.  Yes  or  No 

6.  Present  the  third  section  of  instruction 

7.  Present  the  continuing  motivation  survey 

8.  Present  the  post-test  and  simplified  IMMS  survey 

For  the  motivationally-minimized  version  CAI,  any  motivational  strategies  embedded  in  Osman’s  text, 
Park’s  CAI,  and  the  textbook  were  eliminated.  This  was  to  ensure  that  the  developed  CAI  would  not  have  any 
motivating  features  for  the  students.  However,  through  the  formative  evaluation  with  experts  in  instructional  design, 
strategies  that  were  judged  as  contributing  to  instructional  effectiveness  were  kept  to  maintain  the  inherent  quality  of 
the  instruction.  The  motivationally-minimized  CAI  included  3,017  expository  words  and  210  sentences  with  several 
inserted  technical  drawings.  The  Hupercard  stack  size  was  112  k,  and  it  contained  59  cards. 

The  motivationally-minimized  CAI  was  not  expected  to  be  highly  motivating  to  students  except  in  two 
possible  situations:  when  the  topics  of  the  content  themselves  are  motivating  to  learners,  or  when  the  computer  itself 
may  have  motivating  effects.  However,  these  two  situations  were  not  expected  to  affect  a large  number  of  learners 
because  of  the  technical  nature  of  the  content.  It  was  also  expected  that  there  would  be  a minimal  novelty  effect  of 
using  the  computer  because  of  the  previous  computer-usage  experience  of  learners  as  confirmed  by  their  teachers.  In 
any  case,  any  effects  of  these  types  were  expected  to  be  a constant  across  treatments. 

Motivationally-saturated  CAI  was  developed  from  the  rnotivationally-minimized  version.  Using  the  ARCS 
model,  the  motivational  strategies  presented  by  Keller  & Suzuki  (1988)  and  those  obtained  from  interviews  with  two 
biology  teachers  were  reviewed,  and  all  the  appropriate  ones  (24  strategies  in  total),  as  judged  by  the  researchers, 
were  incorporated  into  instruction  for  enhancing  and  sustaining  attention,  relevance,  and  confidence.  The  criteria 
used  for  the  selection,  based  on  Keller  (1987b),  were  that  motivational  strategies  should: 


not  take  up  too  much  instructional  time, 
not  detract  from  the  instructional  objectives. 

fall  within  the  time  and  money  constraints  of  the  development  and  implementation  aspects  of  the  instruction, 

be  acceptable  to  the  audience, 

be  compatible  with  the  delivery  system, 

be  appropriate  for  the  contents, 

not  take  up  much  CPU  memory, 

be  developed  using  hypercard  2.2, 

not  bother  multiple  learners  in  the  classroom. 
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The  strategies  finally  selected  are  presented  in  Figure  3.  The  motivationally-saturated  CAI  contained  3,306 
expository  words  and  273  sentences.  It  also  included  additional  words  (1,835)  and  sentences  (247)  for  motivational 
strategies,  with  technical  drawings  and  tables  inserted.  The  stack  size  was  393  k,  and  it  contained  107  cards. 

Figure  3.  Selected  Motivational  Strategies 

Attention  Enhancing  Strategies 

AE1  Use  inverse  and  flash  in  text  and  patterns  in  pictures  as  attention  getters 

AE2  Engage  the  learner’s  interest  by  using  question-response-feedback  interaction  that  requires  active  thinking 
AE3  Present  problem  solving  situation  in  a context  of  exploration  and  partial  revelations  of  knowledge 

Attention  Sustaining  Strategies 

ASl  Keep  instructional  segments  relatively  short  with  progressive  disclosure 
AS2  Make  effective  use  of  screen  display  to  facilitate  ease  of  reading 
AS3  Intermingle  information  presentation  screens  with  interactive  screens 
AS4  Use  a consistent  screen  format  but  with  occasional  variation 

AS5  Use  visual  enhancement  functionally  to  support  the  instruction  and  general  theme  of  the  lesson 

AS6  Avoid  dysfunctional  attention-getting  effects  such  as  a flashing  word  that  distracts  learners  concentration 

AS7  Use  underlines,  italics,  or  bigger  font  sizes  for  the  headings  or  key  words 

Relevance  Enhancing  Strategies 

RE1  Use  examples  from  content  areas  and  situation  that  are  familiar  to  the  learners 

RE2  Clearly  state  the  objectives  in  terms  of  the  importance  or  utility  of  the  lesson 

Relevance  Sustaining  Strategies 

RSI  Use  personal  pronouns  and  the  learners’  name  when  appropriate 

RS2  Use  graphic  illustrations  to  embed  abstract  or  unfamiliar  concepts  in  a familiar  setting 

Confidence  Enhancing  Strategies 

CE1  Use  words  and  phrases  that  help  attributes  success  to  the  learners’  effort  and  ability 

CE2  Clearly  present  the  objectives  and  the  overall  structure  of  the  lesson 

CE3  Explain  the  evaluative  criteria  and  provide  opportunities  for  practice  with  feedback 

CE4  Mention  the  prerequisite  knowledge,  skills,  or  attitudes  that  will  help  the  learner  succeed  at  the  task 

CE5  Tell  the  learner  how  many  items  are  going  to  be  in  a test  or  drill,  and  whether  it  will  be  timed. 

CE6  Provide  summary 

CE7  Use  a menu-driven  structure  to  provide  learner  control  over  access  to  different  part  of  the  courseware 
Confidence  Sustaining  Strategies 

CS1  Allow  the  learner  to  escape  and  return  to  the  menu  at  any  time,  and  if  feasible,  to  page  backwards. 

CS2  Give  the  learner  control  over  pacing  by  hitting  a key  to  go  from  one  screen  to  the  next 

CS3  Match  learning  requirements  to  prerequisite  knowledge  and  skills  to  prevent  excessive  challenge  or  boredom. 

One  thing  to  be  mentioned  is  that  motivational  strategies  used  in  this  study  were  classified  into  two 
: categories  of  enhancing  and  sustaining  strategies.  This  distinction  was  necessary  because  Keller  (1983,  1987  a,  b,  c) 
distinguishes  “enhancing  motivation”  from  “sustaining  motivation.”  He  says  that  strategies  for  motivational 
enhancement  are  provided  to  learners  who  are  not  motivated,  while  strategies  for  sustaining  motivation  are  provided, 
as  kind  of  hygiene  factors,  to  learners  who  show  the  desired  levels  of  motivation.  For  example,  in  the  case  of  the 
component  of  ‘attention/  “ animation,  inverse,  flash,  sound  and  other  audio  and  visual  capabilities  of  the  computer”  . 
(from  Figure  3)  were  categorized  as  attention-getting  strategies.  However,  “consistent  screen  format,  but  including 
occasional  variation”  were  categorized  as  attention-sustaining  strategies. 

Because  the  motivationally-saturated  CAI  incorporated  all  the  motivational  strategies  (both  enhancing  and 
sustaining  strategies)  for  all  three  components  (attention,  relevance,  and  confidence),  it  might  seem  that  the 
motivationally-saturated  CAI  would  maximize  learner  motivation.  However,  research  based  on  application  of  the 
ARCS  model  (e.g.,  Visser  & Keller,  1990)  did  not  support  the  excessive  use  of  motivational  strategies.  For  example, 
if  a student  already  feels  confident,  the  confidence  enhancement  strategies  are  not  recommended  to  be  used  in 
instruction.  In  this  study,  the  title,  “motivationally-enhanced  CAI”  was  simply  defined  as  one  which  has 
motivational  strategies  for  both  enhancing  and  sustaining  across  all  three  components:  attention,  relevance,  and 
confidence.  Satisfaction  was  not  considered  here,  as  discussed  earlier  in  this  paper. 

Motivationally-adaptive  CAI  had  all  the  same  features  as  the  motivation  ally- saturated  CAI,  except  that  it 
adaptively  provided  the  learners  only  with  the  appropriate  motivational  strategies.  With  the  motivationally- 
minimized  and  motivationally-saturated  CAI,  all  the  students  were  given  the  same  package  through  the  three  sections 


AVAILABLE 


of  the  instruction,  regardless  of  the  results  of  the  embedded  audience  analyses.  However,  with  the  motivationally- 
adaptive  CAI,  if  a learner  was  analyzed  as  having  low  confidence,  high  attention,  and  high  relevance,  this  version 
provided  the  instruction  which  aimed  to  enhance  and  sustain  confidence,  while  sustaining  the  attention  and  relevance 
level.  More  specifically,  at  the  outset,  learners  were  asked  to  report  their  initial  motivation  (First  Embedded 
Motivational  Analysis).  They  indicated  whether  the  instruction  was  attention-getting,  relevant,  and  appropriately 
challenging.  This  resulted  in  one  of  eight  possible  combinations  of  responses  (see  Column  1 in  Figure  4).  Students 
were  then  given  the  appropriate  combination  of  sustaining  and  enhancing  motivational  strategies  (see  Types  1,  3,  5, 

7,  9,  11,  13,  and  15  in  Column  4 of  Figure  4).  The  feedback  associated  with  each  of  these  types  was  not  given  after 
the  First  Embedded  Motivational  Analysis  as  it  did  not  apply  at  that  time. 

After  reporting  their  motivation  the  second  and  third  times  (Second  and  Third  Embedded  Motivational 
Analysis),  each  student  took  a four-item  quiz.  The  quiz  was  provided  to  check  whether  the  self-reported  confidence 
level  was  consistent  with  the  performance  scores  on  the  quiz.  For  example,  a student  who  reported  high  confidence 
and  obtained  a high  quiz  score  was  considered  to  have  matched  responses,  but  a student  who  reported  high 
confidence  and  had  a low  quiz  score  was  given  a different  type  of  feedback.  The  results  of  this  quiz  were  categorized 
into  two  levels  (see  Column  2 in  Figure  4):  pass  (scores  higher  than  2.5)  vs.  fail  (scores  lower  than  2.5).  The  joining 
of  eight  combinations  of  motivational  responses  and  two  levels  of  quiz  outcomes  resulted  in  a total  of  sixteen  types 
of  student  response  (see  Figure  4,  Column  4).  The  appropriate  type  of  motivational  enhancements  and  feedback  were 
then  implemented  for  each  student. 

Measurement 

Motivation  (both  overall  motivation  and  each  of  attention,  relevance,  confidence,  and  satisfaction)  was 
measured  through  a simplified  version  of  Keller's  (1993)  Instructional  Materials  Motivational  Scale  (IMMS),  which 
measures  the  motivational  features  of  instructional  material  in  terms  of  attention,  relevance,  confidence,  and 
satisfaction.  Internal  consistency  estimates  for  the  IMMS  total  score  and  subscales  are  generally  in  the  range  of  .81  to 
.96.  The  simplified  IMMS  used  for  this  study  had  16  items,  four  items  for  each  component  of  attention,  relevance, 
confidence,  and  satisfaction  respectively.  The  reliability  coefficient,  Cronbach's  Alpha,  for  the  overall  motivation 
measure  was  .92,  and  it  was  .79,  .73,  .70,  and  .85  for  attention,  relevance,  confidence,  and  satisfaction,  respectively. 
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Figure  4 Types  of  Instruction  for  Adaptive  CAI 


A = 
R = 
C = 
Y = 
N = 
NP= 
P = 


Perceived  attention; 

Perceived  relevance 
Perceived  confidence 

Yes,  I have  attention,  relevance,  or  confidence 
No,  I don’t  have  attention,  relevance,  or  confidence 

Enhancing  strategies  are  not  provided  but  sustaining  strategies  are  provided. 


Column  1 

Column  2 

Column  3 

Column  4 

Perceived 

Motivation 

Quiz: 

4 items 

Motivational  Strategies 

Types 

A 

R 

C 

Score 

For 

Attention 

For 

Relevance 

For 

Confidence 

Type 

Y 

Y 

Y 

>2.5 

NP 

NP 

NP 

1 

* Say,  “Very  Good  score!”,  then  provide  the  next 

* If  the  score  is  4,  say  “Excellent  score.” 

topics. 

<2.5 

NP  | 

IN? 1 

|np 1 

J 

♦Say  “Good  try.  Try  hard  to  raise  your  scores.”. 

then  provide  review  + next  topics. 

Y 

Y 

N 

>2.5 

NP  1 

| NP  I P 

! 3 

♦ Say,  “Very  Good  score!  Your  confidence  should  be  growing  stronger.”, 
next  topics  with  confidence  strategies. 

♦ If  the  score  is  4,  say  “Excellent  score.” 

, then  provide  the 

<2.5 

NP  | 

[ NP 

Jp  l 

|4 

♦ Say,  “Good  try.  Try  hard  to  raise  your  scores  and 

confidence  level.”  then,  provide  review  + next  topics  with  confidence  strategies. 

Y 

N 

Y 

> 2.5 

NP  | 

|P : 

1 NP  1 

U 

* Say,  “Very  Good  score!  Was  it  irrelevant  to  you?”,  then,  provide  next  topics  with  relevance 
strategies. 

♦ If  the  score  is  4,  say  “Excellent  score.” 

< IS 

NP  | 

LE 

JN?  1 

\A 

* Say,  “Good  try.  Try  hard  to  raise  your  scores.  Was  it  irrelevant  to  you?”,  then  provide 
review  + next  topics  with  relevance  strategies. 

Y 

N 

N 

>2.5 

NP  | 

IP 

JP  1 

1 ■ 

♦ Say,  “Very  Good  score!  Was  it  irrelevant  to  you?  Your  confidence  should  be  growing 
stronger.”,  then,  provide  next  topics  with  relevance  and  confidence  strategies. 

♦ If  the  score  is  4,  say  “Excellent  score.” 

< 2.5 

NP 

1 p 

JP  1 

Li 

♦ Say,  “Good  try.  Try  hard  to  raise  your  scores  and  confidence  level.  Was  it  irrelevant  to 
you?”  then  provide  review  + next  topics  with  confidence  and  relevance  strategies. 

N 

Y 

Y 

> 2.5 

P 

| NP 

1 NP  . 1 

9 

* Say,  “Very  Good  score!  Was  it  boring  to  you?” 
strategies. 

♦ If  the  score  is  4,  say  “Excellent  score.” 

, then,  provide  the  next  topics  with  attention 

<2.5 

P 

| NP 

1 NP  1 

10 

♦ Say,  “Good  try.  Try  hard  to  raise  your  scores.  Was  it  boring  to  you?”,  then  provide  review 
+ next  topic  with  attention  strategies. 

N . 

Y 

N 

>2.5 

P 

| NP 

|p  i 

nr~ — — 

* Say,  “Very  Good  score!  Your  confidence  should  be  growing  stronger.  Was  it  boring  to 
you?”,  then  provide  next  topics  with  confidence  and  attention  strategies. 

♦ If  the  score  is  4,  say  “Excellent  score.” 

<2.5 

P 

| NP 

|p  i 

112. 

♦ Say,  “Good  try.  Try  hard  to  raise  your  scores  and  confidence  level.  Was  it  boring  to  you?”, 
then  provide  review  + next  topics  with  confidence  and  attention  strategies. . 

-N 

N 

Y 

> 2.5 

P 

_|p 

1 NP  1 

1 13 

* Say,  “Very  Good  score!  Was  it  boring  and  irrelevant  to  you?”,  then,  provide  the  next  topics 
with  attention  and  relevance  strategies. 

* If  the  score  is  4,  say  “Excellent  score.” 

<2.5 

P 

|p 

1 NP  I 

\u 

♦ Say,  “Good  try.  Try  hard  to  raise  your  scores.  Was  it  boring  and  irrelevant  to  you?”,  then 
provide  review  + next  topics  with  attention  and  relevance  strategies. 

N 

-N 

N 

>2.5 

P 

Jp 

p i 

1 15 

♦ Say,  “Very  Good  score!  Your  confidence  should  be  growing  stronger.  Was  it  boring  and 
irrelevant  to  you?”  then  provide  next  topics  with  attention,  relevance,  and  confidence 
strategies. 

♦ If  the  score  is  4,  say  “Excellent  score.” 

N 

N ' 

N 

<2.5 

P 

_|p 

p i 

1 16 

♦ Say,  “Good  try.  Try  hard  to  raise  your  scores  and  confidence  level.  Was  it  boring  and 
irrelevant  to  you?”  then  provide  review  + next  topics  with  attention, 
relevance,  and  confidence  strategies. 
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Learning  achievement  (effectiveness)  was  measured  by  a posttest  of  learners'  understanding  of  the 
instructional  content  (facts,  concepts,  and  principles).  The  test  included  13  multiple-choice  items  which  were 
adopted  and  revised  from  a test  with  24  items  used  in  a prior  study  (Park,  1993).  Park  reported  the  reliability  of  the 
test  as  .78,  using  the  K-R  20  formula.  In  order  for  students  to  complete  all  the  requirements  within  the  experimental 
time,  the  number  of  test  items  was  reduced  to  13  items  with  each  item  being  worth  one  point.  Two  subject  matter 
experts  (biology  teachers)  validated  the  test  according  to  the  learning  objectives  using  a table  of  specifications  for 
the  content  of  instruction.  Cronbach’s  alpha  of  the  test  was  .69.  The  reduction  in  reliability  compared  to  Park’s 
(1993)  could  be  due  to  either  the  reduced  length  of  the  posttest  or  the  differences  in  the  subjects. 

Continuing  motivation  was  measured  by  asking  students  whether  they  wanted  to  learn  more  in  the  future  on 
the  same  or  similar  content  (Figure  2).  Students  were  told  to  report  their  continuing  motivation  on  a Likert-type  item 
with  five  anchors  (1  = Strongly  Disagree,  2 = Disagree,  3 = Not  Sure,  4 = Agree,  5 = Strongly  Agree).  The  item 
asked,  ‘The  lesson  was  motivating  to  me.  I would  like  to  learn  more  about  the  same  subject  matter  later.” 

Efficiency  was  obtained  by  multiplying  1,000  to  the  ratio  of  performance  on  the  post  test  to  study  time.  The 
computer  tracked  the  time  (measured  in  seconds)  used  by  each  student.  For  instance,  if  a student  scored  8 using 
2,000  seconds,  8 was  divided  by  2,000,  and  then  multiplied  by  1,000.  His  efficiency  was  4. 

To  control  for  the  confounding  effect  of  prior  achievement,  students’  scores  in  science  on  the  Metropolitan 
Achievement  Test  7 were  obtained  and  used  in  an  analysis  of  covariance  for  effectiveness  and  efficiency. 


Procedure 

One  month  before  the  experiments,  the  students  were  measured  on  their  pre -motivation  to  learn  genetics. 
This  was  done  to  conduct  audience  motivational  analysis  as  recommended  by  the  ARCS  Model.  The  one-way 
ANOVA  revealed  no  significant  difference  among  randomly  assigned  groups  for  their  motivation  to  learn  genetics  (F 
= .4024,  p =.6706).  Also  a significant  difference  was  not  found  among  groups  on  scores  for  attention  (F  = .3839,  p 
=.6830),  relevance  (F  = .4652,  p = .6304),  confidence  (F  = .6718,  p = .5149),  and  satisfaction  (F  = .0956,  p = .9090). 

The  experiments  were  conducted  at  a learning  resource  center  in  a university  building  near  the  students’ 
school.  Three  80-minute  sessions  (one  session  a day)  were  set  up  in  accordance  with  the  students’  regular  class 
periods.  Students  participated  in  the  one  of  three  motivational  design  conditions  to  which  they  were  randomly 
assigned.  They  were  instructed  to  study  the  materials  with  the  emphasis  that  posttests  and  a survey  would  be  given 
after  their  learning.  The  computer  assignment  and  explanation  took  about  10  to  15  minutes.  They  were  also  told  to 
raise  their  hands  when  they  were  ready  for  the  test.  Then,  learners  were  allowed  to  study  as  long  as  they  wanted,  and 
the  computer  recorded  their  use  of  time.  The  average  time  used  by  students  was  30.49  minutes  (Minimum:  14.6; 

Maxi  mum: 46. 5 2).  When  students  finished  the  lesson,  their  continuing  motivation  was  measured  on  the  computer, 
which  was  followed  by  both  the  post-test  and  the  simplified  IMMS. 

Data  analysis 

To  test  for  differences  in  motivation  (both  overall  motivation  and  each  of  attention,  relevance,  confidence, 
and  satisfaction)  and  continuing  motivation  among  learners  in  three  versions  of  CAI,  one-way  analysis  of  variance 
(ANOVA)  was  used.  To  check  the  results  of  ANOVA  of  the  four  sub-components  of  motivation,  a one-way 
MANOVA  was  conducted.  This  was  done  because  the  sub-components  of  attention,  relevance,  confidence,  and 
satisfaction  showed  significant  correlation’s.  The  MANOVA  results  agreed  with  those  obtained  from  the  ANOVA. 
To  test  for  differences  in  effectiveness  (learning  achievement)  and  efficiency,  one-way  analyses  of  covariance 
(ANCOVA)  were  used  with  students’  science  scores  on  Metropolitan  Achievement  Test  7 as  a covariate.  Both 
ANOVA  and  ANCOVA  were  followed  by  Fisher’s  LSD  pairwise  comparison  procedure  if  a significant  difference 
was  found  among  treatments.  Alpha  was  set  at  .05  for  all  statistical  tests.  All  these  analyses  were  done  with  the 
Statistical  Package  for  the  Social  Science  (SPSS  Windows  V.  6.0). 


Results 


Motivation 

One-way  ANOVA  conducted  on  overall  motivation  scores  revealed  a significant  difference  for  the 
treatments,  F ( 2,  57)  = 4.46,  p < .05.  Approximately  14%  of  the  difference  in  motivation  was  explained  by  the 
treatments.  Observed  power  was  .74.  Fisher’s  LSD  pairwise  comparison  procedures  revealed  that  students  in  the 
motivationally-adaptive  CAI  (M=52.73,  SD=  12.09)  showed  higher  motivation  than  those  in  both  motivationally- 
saturated  (M=43.63,  SD=9.38)  and  motivationally-minimized  CAI  (M=42.84,  SD=  13.75). 

Regarding  attention,  one-way  ANOVA  revealed  a significant  difference  for  the  treatments,  F ( 2,  57)  = 
5.07,  p < .01.  Approximately  15%  of  the  difference  in  attention  was  explained  by  the  treatment,  and  observed  power 
was  .80.  Fisher’s  LSD  pairwise  comparison  procedures  revealed  that  students  in  the  motivationally-adaptive  CAI 
(M=13.36,  SD=3.36)  showed  higher  attention  than  those  in  both  motivationally-saturated  (M=10.95,  SD=3.19)  and 
motivationally-minimized  CAI  (M=  10.00,  SD=3.96). 
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Regarding  relevance,  one-way  ANOVA  revealed  a significant  difference  for  the  treatments,  F ( 2,  57)  = 
4.24,  p < .05.  Approximately  13%  of  the  differences  in  relevance  was  explained  by  the  treatment,  and  observed 
power  was  .72.  Fisher’s  LSD  pairwise  comparison  procedures  revealed  that  students  in  the  motivationally-adaptive 
CAI  (M=  12.50,  SD=3.47)  showed  higher  relevance  than  those  in  motivationally-saturated  CAI  (M=  9.42,  SD=2.97). 
There  was  not  a significant  difference  for  confidence  and  satisfaction. 

Effectiveness 

The  correlation  coefficient,  Pearson  r,  between  science  scores  and  posttest  scores  was  .5402  (p  = .001). 
One-way  ANCOVA  conducted  on  achievement  scores  revealed  a significant  difference  for  the  treatments,  F ( 2,  56) 
= 5.28,  p < .01.  Approximately  16%  of  the  difference  in  achievement  was  explained  by  the  treatments.  Observed 
power  was  .82.  Fisher’s  LSD  pairwise  comparison  procedures  revealed  that  students  in  the  motivationally-adaptive 
CAI  (M=7.18,  SD=2.50)  performed  significantly  better  than  those  in  both  motivationally-saturated  (M=5.84 
SD=3.11)  and  motivationally-minimized  CAI  (M=6.68,  SD=3.35). 

To  investigate  the  validity  of  the  rationale  for  the  assumption  that  increased  motivation  will  produce  more 
effectiveness,  the  correlation  between  overall  motivation  and  adjusted  posttest  scores  was  calculated.  The  observed 
posttest  scores  were  transformed  into  adjusted  scores  for  the  covariate  on  the  validated  assumption  of  homogeneity 
of  regression  coefficient.  The  correlation  coefficient  Pearson  r was  .2546  (p=.05).  This  implies  that  about  6.5%  of 
the  achievement  variance  can  be  explained  by  the  differences  in  overall  motivation  when  the  confounding  effects  of 
the  covariate  are  systematically  taken  out.  This  explained  variance  is  larger  than  the  2.5%  that  was  reported  by 
Bickford  (1989),  who  used  the  ARCS  model  for  printed  material  instruction  non-adaptively. 

Continuing  Motivation 

Although  the  students  in  the  motivationally-adaptive  CAI  (M=3.18,  SD=1.10)  showed  higher  continuing 
motivation  than  both  the  motivationally-saturated  (M=2.42,  SD=1.07)  and  motivationally-minimized  CAI  (M  = 2.63, 
SD  =.96),  one-way  ANCOVA  conducted  did  not  reveal  a significant  difference  for  the  treatments,  F ( 2,  57)  = 2.93, 
p = .06.  Approximately  9%  of  the  difference  in  continuing  motivation  was  explained  by  the  treatment.  Observed 
power  was  . 549. 

Efficiency 

The  correlation  coefficient,  Pearson  r,  between  science  scores  and  the  efficiency  measure  was  .4728 
(p=.001).  One-way  ANCOVA  conducted  on  efficiency  revealed  a significant  difference  for  the  treatments,  F ( 2,  55) 
= 5.17,  p<  .01.  Approximately  16%  of  the  difference  in  efficiency  was  explained  by  the  treatment,  and  observed 
power  was  .81.  Fisher’s  LSD  pairwise  comparison  procedures  revealed  that  students  in  both  the  motivationally- 
adaptive  CAI  (M=4.01,  SD=1.65)  and  motivationally-minimized  CAI  (M=4.62,  SD=2.25)  showed  higher  efficiency 
than  those  in  the  motivationally-saturated  CAI  (M=3.11,  SD=2.11). 
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Table  1 


Means  and  Standard  Deviations  for  Dependent  Variables 


Variables 

Minimized 

CAI  Versions 

Saturated 

Adaptive 

Total 

Motivation1 

n 

19 

19 

22 

60 

Overall 

M 

42.84 

43.63 

52.73 

46.72 

Motivation 

SD 

13.75 

9.38 

12.09 

12.57 

n 

19 

19 

22 

60 

Attention2 

M 

10.00 

10.95 

13.36 

11.53 

3D 

3.96 

3.19 

3.36 

3.74 

n 

19 

19 

22 

60 

Relevance2 

M 

11.00 

9.42 

12.50 

11.05 

SD 

3.64 

2.97 

3.47 

3.56 

n 

19 

19 

22 

60 

Confidence2 

M 

11.95 

12.00 

13.95 

12.70 

SD 

4.02 

2.98 

3.32 

3.53 

n 

19 

19 

22 

60 

Satisfaction2 

M 

10.42 

11.26 

12.95 

11.62 

SD 

4.22 

3.36 

3.92 

3.94 

n 

19 

19 

22 

60 

Effectiveness3  5 

M 

6.68(6.25) 

5.84(5.52) 

7.18(7.94) 

6.60 

SD 

3.35 

3.11 

2.50 

2.99 

n 

19 

19 

22 

60 

Continuing 

M 

2.63 

2.42 

3.18 

2.77 

Motivation4 

SD 

.96 

1.07 

1.10 

1.08 

n 

19 

19 

22 

60 

Efficiency5 

M 

4.62(4.45) 

3.11(2.89) 

4.01(4.42) 

3.91 

SD 

2.25 

2.11 

1.65 

2.05 

*1:  range  for  overall  motivation  (16-80) 

*2:  range  for  attention,  relevance,  confidence,  and  satisfaction  (4-20) 
*3:  range  for  effectiveness  (0-13) 

*4:  range  for  continuing  motivation  (1-5) 

*5:  adjusted  scores  for  a covariate  in  parenthesis 


Discussion 

The  purpose  of  this  study  was  to  investigate  the  effects  of  a motivationally-adaptive  CAI  developed  in 
accordance  with  the  ARCS  model.  For  this  purpose,  a prototypical  motivationally-adaptive  CAI  was  developed  and 
compared  to  a motivationally-saturated  CAI  and  a motivationally-minimized  CAI  for  its  effects  on  motivation 
(overall  motivation,  attention,  relevance,  confidence,  and  satisfaction),  effectiveness,  continuing  motivation,  and 
efficiency. 

Results  indicated  that  motivationally-adaptive  CAI  was  superior  to  the  other  two  CAIs  for  the  enhancement 
of  overall  motivation  and  attention.  Students  in  the  motivationally-adaptive  CAI  may  not  have  been  deprived  of 
motivational  intervention  or  may  not  have  been  overly  manipulated.  Therefore,  the  results  support  the  suggestion 
that  the  amount  of  motivational  emphasis  or  strategies  need  to  be  optimal  rather  than  excessive  or  scarce  (Brophy, 
1987;  Keller,  1983,  1987  a,b,c;  Lepper,  1988;  Suzuki  & Keller,  1998). 

In  fact,  despite  the  emphasis  on  the  optimal  use  of  motivational  strategies  in  the  literature,  few  systematic 
ways  on  “ how  to”  have  been  suggested  for  motivational  enhancement  of  instruction.  Wlodkowski  (1981)  has 
demonstrated  how  to  preplan  to  incorporate  motivations  strategies  into  specific  parts  of  a lesson.  The  ARCS  model 
provides  an  audience  analysis  process  that  facilitates  preplanning  of  motivational  tactics  that  are  adapted  to  learner 
requirements.  However,  in  both  models,  adaptation  of  the  plan  can  be  done  only  in  ins  true  tor- led  settings  and 
depends  on  instructor  insight  to  maintain  the  appropriate  levels.  The  results  of  this  study  suggest  that  the  adaptive 
use  of  motivational  strategies  based  on  direct  student  input  can  also  succeed  and  can  be  incorporated  effectively  into 
computer-assisted  instruction. 

More  specifically,  the  motivationally-adaptive  CAI  was  adaptive  in  two  aspects  that  allowed  for  the  optimal 
provision  of  motivational  strategies.  First,  the  distinction  between  sustaining  strategies  and  enhancing  strategies  was 
a unique  feature  of  the  program.  Through  the  occasional  (three  times)  embedded  audience  analyses,  students  self- 
reported  as  being  motivated  were  provided  only  with  sustaining  strategies,  while  students  diagnosed  as  not  having 
optimal  motivation  were  prescribed  with  enhancing  and  sustaining  strategies.  This  way,  students  were  expected  to 
receive  only  the  exact  amount  of  motivational  intervention  that  was  needed... 
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Second,  the  program  was  adaptive  in  terms  of  appropriate  types  of  motivational  strategies.  Students  were 
diagnosed  for  their  self-perceived  attention,  relevance,  and  confidence  level  based  on  their  “Yes”  or  “No”  responses 
to  each  corresponding  question.  This  came  up  with  8 types  of  student  motivational  profiles.  For  example,  for  the 
first  audience  analysis  at  the  outset  of  instruction,  if  a student  responded  “Yes”  to  attention  and  relevance  questions 
and  “No”  to  the  confidence  question,  he  was  provided  with  instruction  which  included  sustaining  strategies  for 
attention  and  relevance  while  enhancing  and  sustaining  strategies  for  confidence. 

This  kind  of  elaborated  adaptation  was  expected  to  be  more  effective  to  motivate  learners  than  other 
approaches  such  as  those  suggested  by  del  Soldate  & du  Bouley(1995),  who  focused  only  on  confidence,  because  the 
elaborated  adaptation  dealt  with  motivation  more  comprehensively  rather  than  narrowly  focusing  on  specific  aspects 
of  motivation.  In  this  study,  motivation  was  prescribed  for  attention,  relevance,  and  confidence.  Although 
satisfaction  was  not  prescribed  due  to  experimental  constraints,  the  results  support  the  contention  for  comprehensive 
and  adaptive  prescription  that  has  been  lacking  in  the  previous  efforts. 

Regarding  the  other  sub-components  of  motivation,  the  adaptive  use  of  motivational  strategies  was  partly 
supported.  For  relevance,  students  in  the  motivationally-adaptive  CAI  showed  higher  scores  than  the  motivationally- 
saturated  CAI  but  not  the  motivationally-minimized  CAI.  Regarding  confidence  and  satisfaction,  although  students 
in  the  motivationally-adaptive  CAI  showed  the  highest  scores,  significant  differences  were  not  found  among  three 
CAIs. 

These  findings  may  be  explained  as  follows.  In  the  case  of  relevance,  the  fact  that  students  in  the 
motivationally-saturated  CAI  showed  the  lowest  relevance  score  supports  the  contention  that  motivational  strategies 
need  to  be  prescribed  optimally,  i.e.,  adaptively.  In  the  motivationally-saturated  CAI,  strategies  for  all  three 
components,  attention,  relevance,  and  confidence  were  provided  and  distributed  throughout  the  instruction.  This 
means  that  students  had  to  be  exposed  to  much  more  additional  content  than  those  who  were  either  in  the 
motivationally-adaptive  or  motivationally-minimized  CAI. 

For  example,  the  number  of  total  words  used  in  the  motivationally-saturated  CAI  was  5,141,  while  the 
motivationally-minimized  CAI  had  only  3,017  words.  Of  the  difference  (2,124),  1,835  words  were  used  for 
motivational  strategies,  and  this  extra  burden  of  words  might  have  hindered  the  relevance  strategies  from  influencing 
students.  That  is,  students  who  were  distracted  by  all  the  motivational  strategies  may  not  have  noticed  the  relevance 
that  was  intended  to  be  provided  by  a relatively  small  number  of  relevance  strategies.  Furthermore,  when  few 
number  of  relevance  strategies  were  integrated  with  other  strategies,  their  own  effects  could  have  been  reduced. 

Also,  most  of  the  relevance  strategies  that  were  provided  in  the  text  format  may  not  have  been  salient  to  learners 
unless  the  students  had  high  motivational  needs  for  relevance  of  the  contents.  In  contrast,  students  in  the 
motivationally-adaptive  CAI  were  provided  relevance  strategies  only  when  they  were  looking  for  the  usefulness  or 
value  of  the  content. 

The  fact  that  there  was  no  significant  difference  for  relevance  between  the  motivationally-minimized  CAI 
and  the  motivationally-adaptive  CAI  was  interesting  and  drew  the  researchers  attention.  The  question  was  what 
would  have  produced  the  unexpected  relevance  level  in  the  motivationally- minimized  CAI.  One  possible  explanation 
would  be  that  students’  exposure  only  to  the  content,  without  distraction,  possibly  allowed  them  to  recognize  the 
relevance  as  they  learned  more  about  the  content  even  though  they  were  not  given  specific  relevance  strategies.  This 
inference  may  be  supported  by  the  fact  that  students  in  the  motivationally-minimized  CAI  showed  higher 
achievement  than  those  in  the  motivationally-saturated  CAI  who  were  distracted  by  all  the  motivational  strategies. 
However,  the  difference  in  achievement  between  these  two  treatments  was  not  significant,  so  additional  research 
would  have  to  be  done  to  confirm  or  negate  this  inference. 

In  sum,  the  inference  is  that  there  might  have  been  increased  relevance  which  came  from  the  students’ 
achievement  in  the  motivationally- minimized  CAI,  and  its  amount  might  have  been  approximately  the  same  as  the 
small  amount  of  relevance  saturated  in  the  motivationally-adaptive  CAI.  One  possible  rationale  for  this  inference  is 
that  the  average  relevance  score  across  the  three  treatments  was  1 1.05  (SD=3.56),  which  is  at  44  percentile,  below 
the  mid-point  of  the  range  (4-20).  This  implies  that  overall  relevance  reported  across  those  three  treatments  is  quite 
low,  and  possibly  has  floor-effects.  Therefore,  although  the  relevance  level  for  the  motivationally-adaptive  CAI  was 
12.50  (SD=3.47,  at  53  percentile)  which  is  slightly  above  the  average,  it  was  not  significantly  larger  than  that  (1 1.05) 
for  the  motivationally-minimized  CAI.  To  resolve  this,  further  research  would  be  necessary. 

Another  interesting  finding  is  that  there  were  no  differences  in  confidence  among  the  three  conditions.  This 
can  be  interpreted  in  two  ways.  First,  it  is  possible  that  the  confidence  strategies  prescribed  were  not  strong  enough 
to  address  the  motivational  needs.  In  fact,  this  is  a possible  inference  in  that  all  the  powerful  computer  features  could 
not  be  used  due  to  the  restraints  of  experimental  conditions.  Second,  provision  of  review  opportunities  would  have 
resulted  in  non-significant  differences  in  confidence  among  the  three  conditions.  Research  results  show  that  review 
opportunities  are  effective  strategies  for  increasing  students’  confidence  (Bickford,  1989).  Therefore,  review 
opportunities  that  were  provided  before  and  after  taking  each  quiz  may  have  evenly  contributed  to  maintaining  or 
enhancing  learners’  confidence  across  the  three  conditions.  Further  investigation  of  this  inference  needs  to  be  done  in 
future  studies. 
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Regarding  satisfaction,  there  were  no  differences  among  the  three  treatments.  However,  this  finding  did  not 
get  the  researchers'  attention  because  each  CAI  version  was  designed  not  to  contain  satisfaction  strategies.  Without 
systematic  manipulation  of  satisfaction,  students  may  have  provided  their  own  satisfaction  on  a random  basis. 

Aside  from  the  motivational  benefits  of  the  adaptive  use  of  the  ARCS  model,  another  expected  benefit  is  to 
increase  achievement  scores.  Although  motivation  to  learn  is  not  the  only  predictor  of  student  achievement,  it 
seemed  reasonable  to  predict  that  if  motivationally-adaptive  CAI  is  more  motivating  than  the  other  two  CAIs,  it 
would  also  show  the  highest  effectiveness.  The  results  of  this  study  supported  this  expectation. 

Regarding  continuing  motivation,  Keller  (1983,  1987a,  b)  points  out  that  the  more  highly  motivated  the 
learners  are  during  their  learning,  the  better  chance  of  their  having  high  continuing  motivation  after  their  learning. 
However,  although  students  in  the  motivationally-adaptive  CAI  showed  the  highest  continuing  motivation,  the 
amount  was  not  enough  to  be  significantly  different  from  those  of  students  in  the  other  two  CAIs.  Considering  the 
positive  test  results  for  overall  motivation,  the  motivationally-adaptive  CAI  should  have  produced  more  continuing 
motivation  than  the  other  two  CAIs. 

One  interpretation  of  this  inconsistent  finding  is  that  motivationally  enhanced  treatments  is  not  the  sole 
predictor  of  continuing  motivation.  If  other  confounding  factors  influenced  continuing  motivation,  it  would  be 
difficult  to  have  significant  differences  across  the  three  conditions.  Also,  the  observed  power  for  ANOVA  on 
continuing  motivation  was  .549.  This  shows  that  a larger  sample  size  would  have  revealed  the  significant  differences 
among  the  three  conditions,  and  a longer  treatment  may  also  be  beneficial. 

The  above  interpretation  is  also  indirectly  evidenced  by  the  correlation  (Pearson  r,  .6573,  p=.000)  between 
overall  motivation  and  continuing  motivation,  although  not  all  of  the  treatments  conditions  had  a significant 
correlation  with  continuing  motivation..  The  motivationally-adaptive  CAI  showed  the  highest  correlation  (.8155,  p= 
.0000),  while  the  motivationally-minimized  CAI  was  also  significant  (r=.6424  (p  = .003).  It  is  interesting  that  the 
motivationally-saturated  CAI  showed  the  lowest  correlation,  .3426,  which  was  not  significant  (p=.151).  These 
correlation's  may  indicate  that  the  motivationally-adaptive  CAI  has  more  chance  of  increasing  both  motivation  and 
continuing  motivation  than  the  other  two  CAIs.  Why  the  motivationally-saturated  CAI  did  not  show  significant 
correlation  between  motivation  and  continuing  motivation  should  be  further  investigated. 

For  efficiency,  it  was  expected  that  learners  in  the  motivationally-adaptive  CAI  would  need  less  time  to 
finish  their  learning  than  those  who  had  to  go  through  all  the  motivational  strategies  in  the  motivationally-saturated 
CAI.  Also,  it  was  expected  that  the  motivationally-minimized  CAI  would  require  less  time  to  finish  than  the  other 
conditions  because  no  motivational  strategies  were  provided.  The  ANCOVA  on  efficiency  showed  that  there  are 
differences  across  the  three  treatments,  and  Fisher's  LSD  analysis  revealed  that  the  motivationally-adaptive  and 
motivationally-minimized  CAIs  showed  higher  efficiency  than  the  motivationally-saturated  CAI.  These  findings 
support  the  need  for  providing  motivational  strategies  adaptively  for  efficiency.  The  excessive  provision  of 
motivational  strategies  in  the  motivationally-saturated  CAI  resulted,  as  expected,  in  lower  efficiency  than  the 
motivationally-minimized  CAI,  while  the  motivationally-adaptive  CAI  showed  the  same  efficiency  with  the 
motivationally-minimized  CAI. 

Overall,  the  idea  of  motivationally-adaptive  CAI  was  empirically  supported  by  the  data  in  terms  of  its 
effectiveness,  efficiency,  and  overall  motivation.  The  data  also  indicated  that  there  is  a significant  correlation 
between  motivation  and  continuing  motivation.  These  findings  support  the  adaptive  provision  of  motivational 
strategies  in  CAI. 

One  theoretical  implication  of  these  findings  is  that  motivation  to  learn  can  be  a useful  indicator  of  learning 
readiness.  Indicators  of  learning  readiness  used  in  previous  studies  in  adaptive  CAI  have  been  mostly  cognitive 
variables  (e.g.,  Atkinson,  1976;  Hativa  & Lesgold,  1991;  Houlihan,  Finkelstein,  & Johnson,  1992;  Mills  & Ragan, 
1994;  Ross  & Morrison,  1988;  Tennyson  & Christen,  1988;  Tennyson  & Park,  1980).  However,  the  results  of  this 
study  support  the  contention  that  motivation  could  be  a strong  predictor  of  learners'  future  performance  and 
motivation  because  students'  motivation  at  one  point  of  time  is  a reflection  of  all  their  previous  experience  in  both 
affective  and  cognitive  domains.  That  is,  students'  previous  attitudes,  values,  learning  style,  IQ,  GPA,  abilities,  and 
aptitudes  have  influence  on  their  current  motivation  to  which  motivational  strategies  can  be  prescribed  adaptively. 

As  an  initial  attempt  to  design  and  develop  a motivationally-adaptive  CAI,  this  study  has  limitations  that 
might  reduce  the  generalizability  of  the  results  and  conclusions.  First,  the  potential  range  of  motivational  strategies 
could  not  be  fully  actualized  due  to  constraints  of  available  hardware  and  software.  For  example,  after  instruction, 
many  students  commented  on  the  lack  of  colors,  animation,  and  sounds  for  more  motivational  features.  Some 
important  conditions  (Astleitner  & Keller,  1995)  for  motivated  learners  such  self-created  task  solution,  opportunities 
for  advanced  topics,  and  control  over  important  parts  of  the  learning  environment  were  not  used. 

Second,  the  time  for  the  experiment  was  short.  The  average  time  was  30.49  minutes  across  three  conditions 
(motivationally-minimized  CAI:  24.37,  saturated  CAI:  35.45,  adaptive  CAI:  31.21).  Considering  the  limited  use  of 
motivational  strategies  as  discussed  above,  this  short  term  exposure  to  motivational  strategies  could  have  been 
another  reason  for  showing  several  insignificant  differences  despite  the  overall  positive  results  that  were  found. 
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Third,  the  use  of  self-report  methods  for  measuring  motivation  were  limited  in  that  they  required  students  to 
indicate  their  perceived  motivation  level,  which  might  have  been  different  from  their  actual  amount  of  effort  which  is 
a more  accurate  measure  of  motivational  behavior.  Also,  the  embedded  motivational  analyses  may  have  been 
intrusive,  requiring  students  to  stop  their  process  of  learning.  A more  natural  way  of  measuring  motivation  would  be 
desirable. 

Fourth,  too  much  information  presented  within  a short  period  of  time  may  have  caused  cognitive  overload 
which  reduced  the  achievement  and  motivation  across  all  three  conditions.  Astleitner  & Keller  (1995)  pointed  out 
that  an  excess  of  information  makes  it  more  difficult  for  the  learner  to  maintain  a sense  of  control  and 
comprehensibility  in  the  learning  environment. 

Fifth,  motivational  strategies  used  in  this  study  may  have  not  addressed  the  real  motivational  needs  of 
students,  resulting  in  fewer  effects  than  expected.  For  example,  students  reported  their  motivational  state  for  each 
component  of  attention,  relevance,  and  confidence,  and  motivational  strategies  were  prescribed  based  on  this 
information.  However,  there  might  be  a need  to  focus  on  the  sub-categories  of  each  component  for  more  reliable 
diagnosis  and  prescription.  For  example,  confidence  is  composed  of  the  sub-components  of  motive  matching,  goal 
orientation,  and  familiarity.  If  motivational  analysis  can  be  done  for  each  of  these  sub-components,  adaptive  CAI 
might  be  more  effective  in  enhancing  students’  confidence. 

Despite  the  limitations  in  this  prototype  development  study,  there  was  overall  support  for  the  adaptive 
treatment,  and  there  are  several  areas  that  would  be  useful  for  continued  study.  First,  this  study  showed  that  about 
14%  of  perceived  motivation  was  explained  by  treatments  with  limited  use  of  computer  features.  It  is  recommended 
that  future  studies  will  utilize  more  powerful  computer  features  such  as  sound,  color,  animation,  video,  and 
interaction.  The  motivationally-adaptive  CAI  could  then  be  even  more  motivating  to  learners. 

Second,  it  is  recommended  to  elaborate  the  ARCS  model  or  to  devise  new  models  which  can  be  used  for  the 
development  of  motivationally-adaptive  CAI.  Although  this  study  pioneered  the  use  of  the  ARCS  model  for  the 
development  of  motivationally-adaptive  CAI,  there  is  a need  for  further  development  of  this  or  other  models  that  can 
be  used  for  the  design  and  development  of  motivationally-adaptive  courseware  for  computer,  multimedia,  or 
network-based  learning  environments  (Astleitner  & Keller,  1995).  A future  elaboration  of  the  model  should 
incorporate  a more  detailed  and  transparent  motivational  diagnosis  system  and  a more  reliable  prescription  system 
with  a database  of  situation-specific  motivational  strategies.  For  example,  motivational  strategies  for  conventional 
CAI  may  be  different  from  those  for  multimedia  learning  environments,  which  utilize  multi-sensory  effects. 

Third,  research  is  necessary  to  investigate  the  relationship  among  pre-motivation,  prior  achievement, 
cognitive  aptitude,  perceived  motivation,  computer  attitude,  and  continuing  motivation  in  motivationally-adaptive 
CAI  in  comparison  with  non-adaptive  CAI.  Continuing  research  on  these  issues  will  provide  the  foundation  that  will 
foster  research  efforts  for  designing  and  developing  a more  advanced  motivationally-adaptive  CAI. 

In  conclusion,  the  results  of  this  study  indicate  the  possible  value  of  the  ARCS  model  as  a guiding  process 
for  designing  motivationally-adaptive  CAI.  It  also  demonstrates  that  self-reports  of  motivation  can  be  a valid 
indicator  of  learning  readiness  to  which  motivational  strategies  are  adaptively  prescribed.  And,  the  study 
demonstrates  that  it  is  feasible  to  design  motivationally-adaptive  instruction  for  a self-paced  learning  setting,  such  as 
CAI.  Further  research  should  lead  to  more  sophisticated  and  effective  applications  of  motivationally  adaptive  design. 

References 

Astleitner,  J.  & Keller,  J.M.  (1995).  A model  for  motivationally  adaptive  computer-assisted  instruction. 
Journal  of  Research  on  Computing  in  Education.  27(3),  270-80. 

Atkinson,  R.C.  (1976).  Adaptive  instructional  systems:  Some  attempts  to  optimize  the  learning  process. 

In  D.  Klahr  Ed.  Cognition  and  Instruction,  (pp.81-108)  New  York:  Wiley  & Sons. 

Bickford,  N.L.  (1989).  The  systematic  application  of  principles  of  motivation  to  the  design  of  printed 
instructional  materials.  Unpublished  doctoral  dissertation.  Florida  State  University,  Tallahassee. 

Brophy,  J.E.  (1987).  On  motivating  students.  In  D.  Berliner  & B.  Rosenshine  (Eds.),  Talks  to  teacher. 
(pp.201-245).  New  York:  Random  House 

Brown,  J.  (1986).  Some  motivational  issues  in  computer-based  instruction.  Educational  Technology, 

26(4),  27-29. 

Carrier,  C.A.  & Jonassen,  D.H.  (1988).  Adapting  courseware  to  accommodate  individual  differences.  In 
Jonassen,  D.H.  (Ed.)  Instructional  designs  for  microcomputer  courseware,  (pp.  203  - 245).  Lawrence  Erlbaum 
Associates,  NJ. 

Clark,  R.E.  (1983).  Reconsidering  research  on  learning  from  media.  Review  of  Educational  Research, 

53(4), 445-459. 


527 


best  copy  available 


498 


del  Soldato,  T.  & du  Boulay,  B.  (1995).  Implementation  of  motivational  tactics  in  tutoring  systems. 
Journal  of  Artificial  Intelligence  in  Education,  6(4),  337-378. 

Dempsey,  J.,  Lucassen,  B.,  Gilley,  W.,  & Rasmussen,  K.  (1993).  Since  Malone's  theory  of  intrinsically 
motivating  instruction:  What's  the  score  in  the  gaming  literature?  Journal  of  Educational  Technology  Systems, 
22(2),  173-184. 

Gagne,  R.M.  (1985).  The  conditions  of  learning  and  theory  of  instruction.  (4th  ed.).  New  York:  Holt, 
Rinehart  & Winston. 

Goodman,  H.D.  et  al.  (1989).  Biology  Orlando,  FL:  Harcourt  Brace  Jovanovich. 

Hativa,  N.,  & Lesgold,  A.  (1991).  The  computer  as  a tutor— can  it  adapt  to  the  individual  learner? 
Instructional  Science,  20(1),  49-78. 

Holland,  J.G.  (1977).  Variables  in  adaptive  decisions  in  individual  instruction.  Educational  Psychologist, 
12(2),  146-161. 

Houlihan,  P.A.,  Finkelstein,  M.W.,  & Johnson,  L.A.  (1992).  Adaptive  use  of  the  DDx  & Tx  system. 
Journal  of  Computer-Based  Instruction,  19(4),  125-130. 

Johnson,  D.W.,  & Johnson,  R.T.  (1986).  Computer-assisted  cooperative  learning.  Educational 
Technology,  26(1),  12-18. 

Keller,  J.M.  (1979).  Motivation  and  instructional  design:  A theoretical  perspective.  Journal  of 
Instructional  Development.  2(4),  26-34. 

Keller,  J.M.  (1983).  Motivational  design  of  instruction.  In  C.M.  Reigeluth  (Ed.),  Instructional-design 
theories  and  models:  An  overview  of  their  current  status  (pp.  386-434).  Hillsdale,  NJ:  Lawrence  Erlbaum 
Associates. 

Keller,  J.M.  (1987a).  Strategies  for  stimulating  the  motivation  to  learn.  Performance  and  Instruction 
Journal,  26(8),  1-7. 

Keller,  J.M.  (1987b).  The  systematic  process  of  motivational  design.  Performance  and  Instruction 
Journal,  26(9/10).  1-8. 

Keller,  J.M.(  1987c).  Development  and  use  of  the  ARCS  model  of  instructional  design.  Journal  of 
Instructional  Development,  10(3),  2-10. 

Keller,  J.M.  (1993).  Motivation  by  design.  Unpublished  manuscript,  Florida  State  University, 
Tallahassee,  Florida. 

Keller,  J.M.  (1997).  Motivational  design  and  multimedia:  Beyond  the  novelty  effect.  Strategic  Human 
Resource  Development  Review,  1(1).  188-203. 

Keller,  J.M.  & Keller,  B.H.  (1991).  Motivating  learners  with  multimedia  instruction.  Proceedings  of 
International  Conference  on  Multi-Media  in  Education  and  Train ing(ICOMMET  ’91),  Tokyo,  Japan:  The 
Japanese  Association  for  Educational  Technology  and  the  International  Society  for  Technology  in  Education. 


Keller,  B.H.,  & Keller,  J.M.  (1992).  Motivating  interactivity  in  multimedia  (The  MIM  checklist).  In  J.M. 
Keller,  Motivation  by  Design,  1995,  Tallahassee,  FL:  John  Keller  Associates. 

Keller,  J.  M.  & Suzuki,  K.  (1988).  Use  of  the  ARCS  motivation  model  in  courseware  design.  In  D.  H. 
Jonassen(Ed.),Instructional  designs  for  microcomputer  courseware  (pp.  401-434).  Hillsdale,  NJ:  Lawrence 
Erlbaum  Associates. 

Klein,  J.  D.  & Freitag,  E.  T.  (1992).  Training  students  to  utilize  self-motivational  strategies.  Educational 
Technology,  32(3),  44-48. 

Klein,  J.D.  & Keller,  J.M.  (1990).  Influence  of  student  ability,  locus  of  control,  and  type  of  instructional 
control  on  performance  and  confidence.  Journal  of  Educational  Research,  83(3),  140-146. 

Lee,  S.  H.  & Boling,  E.  (1996).  Motivational  screen  design  guidelines  for  effective  computer-mediated 
instruction.  Paper  presented  at  the  Association  for  Educational  Communications  and  Technology  (AECT) 

Annual  Conference,  Indianapolis. 


313-316. 


528 


Lepper,  M.R.  (1985).  Microcomputers  in  education:  Motivational  and  social  issues,  American 
Psychologist,  40(1),  1-18. 

Lepper,  M.R.  (1988).  Motivational  considerations  in  the  study  of  instruction.  Cognition  and  Instruction, 
5(4),  289-309. 

Maehr,  M.L.  (1976).  Continuing  motivation:  An  analysis  of  a seldom  considered  educational  outcome. 
Review  of  Educational  Research,  46(3),  443-462. 

Malone,  T.  (1981).  Toward  a theory  of  intrinsically  motivation  instruction.  Cognitive  Science,  4,  333- 

369. 


Malouf,D.B.  (1988).  The  effect  of  instructional  computer  games  on  continuing  student  motivation.  The 
Journal  of  Special  Education,  2J.(4),  27-38. 

McCombs,  B.L.,  Eschenbrenner,  Jr.,  A.J.,  & O’Neill,  Jr.,  H.F.  (1973).  An  adaptive  model  for  utilizing 
learner  characteristics  in  computer  based  instructional  systems.  Educational  Technology,  75(4),  47-51. 

Means,  T.B.,  Jonassen,  D.H.,  & Dwyer,  F.M.  (1997).  Enhancing  relevance:  Embedded  ARCS  strategies 
vs.  purpose.  Educational  Technology  Research  and  Development,  45(1),  5-18. 

Mills,  S.C.  & Ragan,  T.J.  (1994).  Adapting  instruction  to  individual  learner  differences:  A research 
paradigm  for  computer-based  instruction.  Proceedings  of  Selected  Research  and  Development  Presentations  at 
the  1994  Annual  National  Convention  of  the  Association  for  Educational  Communications  and  Technology. 

16th,  Nashville,  TN,  February  16-20. (ERIC  Document  Reproduction  Service  No.  ED  373  740). 

Newby,  T.J.  (1991),  Classroom  motivation:  Strategies  of  first-year  teachers.  Journal  of  Educational 
Psychology,  83(2),  195-200. 

Nwagbara,  C.I.  (1993).  Effects  of  the  relevance  component  the  ARCS  model  of  motivational  design. 
Unpublished  doctoral  dissertation.  Purdue  University. 

Osman,  M.  (1992).  The  effects  of  think-ahead  questions  and  prior  knowledge  on  learning  and  retention. 
Unpublished  Doctoral  Dissertation,  Florida  State  University,  Tallahassee,  FL. 

Park,  I.W.  (1993).  The  effects  of  orienting  questions  and  prior  knowledge  on  learning  in  hypertext. 
Unpublished  doctoral  dissertation,  Florida  State  University,  Tallahassee. 

Park,  O.  (1996).  Adaptive  instructional  systems.  In  Jonassen,  D.  (Ed.).  Handbook  of  research  for 
educational  communications  and  technology,  (pp.  634-664),  New  York:  Simon  & Schuster  Macmillan. 

Relan,  A.  (1992).  Motivational  strategies  in  computer-based  instruction:  Some  lessons  from  theories  and 
models  of  motivation.  In  Proceedings  of  Selected  Research  and  Development  Presentations  at  the  Convention  of 
the  Association  for  Educational  Communications  and  Technology  (ERIC  Document  Reproduction  Service  No. 
ED  348  017). 

Rezabek,  R.H.  (1994).  Utilizing  intrinsic  motivation  in  the  design  of  instruction.  In  Proceedings  of 
Selected  Research  and  Development  Presentations  at  the  Convention  of  the  Association  for  Educational 
Communications  and  Technology  (ERIC  Document  Reproduction  Service  No.  ED  373  751). 

Ross,  S.M.,  & Morrison,  G.R.  (1988).  Adapting  instruction  to  learner  performance  and  background 
variables.  In  D.H.  Jonassen  (Ed.).  Instructional  designs  for  microcomputer  courseware,  (pp.  227-243).  Hillsdale, 
NJ:  Lawrence  Erlbaum  Associates. 

Ross,  S.M.,  Rakow,  E.A.,  & Bush,  A.J.(1980).  Instructional  adaptation  for  self-managed  learning 
systems.  Journal  of  Educational  Psychology,  12(3),  312-320. 

Small,  R.  V.  & Gluck,  M.  (1994).  The  relationship  of  motivational  conditions  to  effective  instructional 
attributes:  A magnitude  scaling  approach.  Educational  Technology.  34(8),  33-40. 

Tennyson,  R.D.  & Christensen,  D.L.  (1988).  MAIS:  an  intelligent  learning  system.  In  D.  Jonassen,  (Ed.). 
Instructional  designs  for  micro  computer  courseware,  (pp.  247-74).  Hillsdale,  NJ:  Erlbaum 

Tennyson,  R.D.,  & Park,  S.  (1984).  Process  learning  time  as  an  adaptive  design  variable  in  concept 
learning  using  computer-based  instruction.  Journal  of  Educational  Psychology,  76,  452-65. 

Tuckman,  B.W.  (1994).  Conducting  educational  research  (4th  ed.).  New  York:  Harcourt  Brace  & 
Company. 


529 


500 


Visser,  J.  & Keller,  J.  M.  (1990).  The  clinical  use  of  motivational  messages:  An  inquiry  into  the  validity  of 
the  ARCS  model  of  motivational  design.  Instructional  Science,  19,  467-500. 

Wlodkowski,  R.J.  (1981).  Making  sense  out  of  motivation:  A systematic  model  to  consolidate 
motivational  construct  across  theories.  Educational  Psychologist.  16(2),  101-110. 


501 


ERIC 


530 


International  Team  Teaching:  A Partnership  Between  Mexico  and  Texas 


Lauren  Cifuentes 
Karen  L.  Murphy 
Texas  A&M  University 


Abstract 

In  this  case  study,  we  report  on  collaborative  activities  among  four  fourth  grade  teachers  and  their 
students  over  a year  and  explore  the  effectiveness  of  distance  learning  for  international  team  teaching.  Two 
classes  in  Mexico  City  conducted  school  activities  with  two  classes  in  College  Station , Texas.  We  asked — 
What  are  some  school  activities  that  successfully  bring  children  of  Mexican  and  Texan  cultures  together  to 
learn  with,  about,  and  from  each  other?  What  similarities  and  differences  in  language  skills  and  pastimes 
exist  between  the  children  of  both  cultures?  What  prevailing  impressions  did  Mexican  and  Texan  children 
have  of  each  other  prior  to  their  distance  learning  activities?  The  activities  in  the  partnership  were 
designed  to  help  the  students  understand  each  other's  cultures  by  walking  in  each  other’s  footsteps. 
Mexican  children  had  enough  experience  of  U.S  culture  to  be  able  to  accurately  describe  the  culture  in 
poetry.  On  the  other  hand,  most  Texan  children  had  little  knowledge  of  Mexico  and  had  to  write  about  their 
own  culture,  create  an  imaginary  place,  or  refer  to  stereotypes  in  order  to  create  a poem. 

Background  and  Theoretical  Perspective 

In  this  case  study,  we  report  on  collaborative  activities  among  four  fourth  grade  teachers  and  their  students 
over  a year  and  explore  the  effectiveness  of  distance  learning  for  international  team  teaching.  Two  classes  in  Mexico 
City  conducted  school  activities  with  two  classes  in  College  Station,  Texas.  In  a previous  study,  diverse  students 
collaborated  across  Texas  on  multicultural  activities,  which  helped  them  grow  in  self-esteem  and  multicultural 
understanding  (Cifuentes,  Murphy,  & Davis,  1998).  The  cross-classroom  collaboration,  Cultural  Connections,  made 
it  possible  for  students  to  expand  their  world-views  in  preparation  for  contributing  in  our  increasingly  multicultural 
environment.  The  previous  study  demonstrated  that  in  networked  classrooms  students  can  connect  with  distant  others 
to  leam  about  and  from  their  perspectives  and  to  increase  their  multicultural  understanding.  In  addition,  distance 
technologies  can  foster  team  teaching  across  cultures  and  geographical  distances  within  Texas.  Four  overarching 
themes  emerged  from  the  data:  growth,  empowerment,  comfort  with  technology,  and  mentoring.  These  themes 
permeated  each  data  source  and  applied  to  all  participants  of  the  Texas  project.  Participants  grew  personally  and 
intellectually.  They  felt  empowered  to  achieve  goals.  They  became  comfortable  with  technology,  and  they  provided 
and/or  received  mentoring.  Students  mentored  and  learned  from  each  other.  They  also  had  the  benefit  of  receiving 
mentoring  from  both  their  local  and  distant  teachers.  We  expect  that  the  international  Cultural  Connections  project 
will  have  similar  yet  unique  effects. 

Distance  technologies  expand  the  range  of  opportunities  for  students  to  build  relationships  with  people  of 
different  cultures.  In  the  current  social-constructivist  conception  of  learning,  "education  is  the  shared  way  of  thinking 
about  one's  self,  the  community,  and  the  world"  (Riel,  1995,  p.  219).  Therefore,  schools  might  play  a significant  role 
in  nurturing  students'  positive  identity  formation,  spirit  of  community,  and  multicultural  perspectives  by  providing 
them  with  opportunities  to  build  distant  relationships.  In  order  to  become  more  tolerant  and  respectful  citizens, 
students  need  to  develop  relationships  with  people  from  diverse  cultures  and  backgrounds  (Moffett,  1994). 
Geographical  and  cultural  isolation  can  limit  opportunities  for  relationship  building  beyond  one's  culture.  However, 
cross-classroom  collaboratives  allow  students  to  connect  with  distant  others  from  around  the  world.  The 
collaborative  learning  process  has  the  potential  to  transform  individual  participants’  perspectives  from  parochial  to 
global  (Cummins  & Sayers,  1995). 

In  the  current  study  we  applied  the  Cultural  Connections  model  with  an  international  team  of  teachers. 
Predominantly  Mexican  students  in  Sierra  Nevada  Bilingual  School  in  Mexico  City  collaborated  with  diverse 
students  in  Rock  Prairie  Elementary  School  in  College  Station,  Texas,  primarily  via  interactive  videoconference  with 
occasional  email.  To  facilitate  building  a broad  world-view  in  their  students,  educators  provided  collaborative 
learning  experiences  for  social  construction  of  meaning.  Telecommunications  were  used  to  expand  the  range  of 
exposure  to  multicultural  interactions  (Cummins  & Sayers,  1995). 


Research  Questions 

Participating  teachers  conducted  at  least  six  distance  learning  experiences  with  their  students  over  a school 
year.  They  planned  the  shared  units  of  study  using, telecommunications.  In  this  study  we  asked — 


1 . What  are  some  school  activities  that  successfully  bring  children  of  Mexican  and  Texan  cultures  together  to  leam  with, 
about,  and  from  each  other? 
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2.  What  similarities  and  differences  in  language  skills  and  pastimes  exist  between  the  children  of  both  cultures? 

3.  What  prevailing  impressions  did  Mexican  and  Texan  children  have  of  each  other  prior  to  their  distance  learning  activities? 


Methods 

Four  teachers,  two  in  Mexico  City  and  two  in  College  Station,  TX,  partnered  for  planning  and 
implementation  of  curricular  activities  with  their  students.  The  forty-one  fourth  grade  students  in  two  classes  in 
College  Station  and  the  forty-six  fourth  grade  students  in  two  classes  in  Mexico  City  were  active  in  the  partnership 
throughout  the  academic  year.  The  Mexican  school  was  a private  school  that  met  in  a converted  home  in  an 
exclusive  neighborhood  of  the  city.  The  Texan  school  was  public.  The  school  partners  in  both  Texas  and  Mexico 
City  were  equipped  with  interactive  compressed  video  systems  at  local  distance  learning  centers,  multimedia 
software,  and  Internet  connections.  Project  administrators  in  both  Texas  and  Mexico  helped  with  planning  and 
facilitation  of  connections.  As  in  the  Texas  partnerships,  several  of  the  curricular  activities  were  stimulated  by  the 
book  / Felt  Like  / Was  From  Another  Planet  (Dresser,  1994).  For  example,  in  a pilot  study,  the  students  conducted 
an  activity  on  table  manners  by  acting  out  and  discussing  the  table  manners  typical  of  their  own  cultures.  In  the  main 
study,  teachers  conducted  ongoing  activities  between  September  and  June. 

The  investigation  relied  on  arts-based  case  study  methods  We  conducted  content  analyses  of  each  of  several 
data  sources:  (a)  lesson  plans  and  observations  of  lesson  implementation,  (b)  students’  poems  about  each  others’ 
nations,  (c)  students’  stories  about  a day  in  the  life  of  a child  in  the  other  culture,  (d)  student  questionnaires,  (e) 
student  collages,  and  (f)  student  self-profiles.  We  looked  for  emergent  themes  and  gained  consensus  on  the  extent  to 
which  the  data  sources  revealed  answers  to  the  research  questions.  As  a result,  we  provide  rich  description  of  the 
project  in  order  to  answer  the  research  questions. 

Findings  and  Discussion 

Question  1:  What  are  some  school  activities  that  successfully  bring  children  of  Mexican  and  Texan 
cultures  together  to  learn  about  each  other? 

The  activities  in  the  partnership  were  designed  to  help  the  students  understand  each  other’s  cultures  by 
walking  in  each  other’s  footsteps.  First,  all  four  classes  attended  poetry  writing  workshops  in  which  they  wrote 
poems  about  the  other  nation.  Though  many  students,  particularly  the  Texans,  had  only  a vague  concept  of  the  other 
nation,  they  were  asked  to  pretend  that  they  were  the  others’  nation  and  to  write  an  “I  am...”  poem  of  images  and 
activities  that  define  that  nation.  Secondly,  the  students  wrote  a story  of  a day  in  the  life  of  a fourth  grader  in  the 
others’  country.  In  addition,  they  filled  out  a survey  that  provided  some  indication  of  the  values  and  artistic  interests 
of  each  child  (see  Appendix  A).  They  also  created  a student  profile  sheet  with  their  picture  on  it  that  described  their 
families,  favorite  subjects  in  school,  favorite  foods,  and  special  interests.  They  shared  these  poems,  stories,  surveys, 
and  profiles  with  each  other  during  the  school  year. 

The  classes  met  twice  for  one  and  one-half  hours  of  videoconferencing  in  which  they  introduced 
themselves,  showed  collages  of  images  that  represented  their  family  and  interests,  and  shared  personal  ancestry.  In  a 
third  videoconference  they  conducted  the  activity  described  below: 

The  Alamo:  A Distance  Learning  Activity 

Objectives: 

As  a result  of  participating  in  the  activities  students  will  be  able  to— 

1 . describe  why  the  battle  of  the  Alamo  occurred. 

2.  create  a dramatic  representation  of  the  battle. 

3.  write  an  essay  comparing  the  U.S.  and  Mexican  interpretations  of  the  story  of  the  Alamo. 

Resources: 

1 . Brief  story  of  the  Alamo  without  mention  of  the  names  of  Mexico,  the  U.S.  or  the  Alamo  (see  Appendix  B). 

2.  Chapters  covering  the  story  of  the  Alamo  from  the  Texas  history  text  and  from  the  Mexican  history  text. 

Activities: 

Preliminary  to  the  videoconference: 

1.  Students  read  a short  story  associated  with  the  story  behind  the  battle  of  the  Alamo.  The  names  Mexico,  the  U.S.,  and  the 
Alamo  are  changed  so  that  students  can  read  objectively. 

2.  Students  pretend  that  they  are  a person  at  the  battle.  They  choose  a side  that  they  are  on  and  write  a diary  entry  about  what 
they  do  and  what  happens  to  them. 

3.  The  teacher  tells  students  that  the  facts  they  read  are  about  the  Alamo  of  1836. 

4.  Students  read  chapters  from  the  Texas  text  and  the  Mexican  text  and  conduct  supportive  reading  activities. 

5.  The  whole  class  identifies  similarities  and  differences  between  the  stories  told  in  the  texts. 

6.  Each  student  writes  a comparative  essay  regarding  interpretations  of  the  battle  of  the  Alamo. 
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7.  In  small  groups  or  as  a whole  class,  students  develop  a brief  (10  minute)  reenactment  of  a story  about  the  Alamo. 

8.  Teacher  selects  one  (or  more)  short  stories  for  presentation  to  the  distant  audience. 

9.  Teacher  selects  one  (or  more)  comparative  essay  for  presentation  to  the  distant  audience. 

1 0.  Teacher  selects  one  reenactment  for  presentation  to  the  distant  audience. 

Activities  during  the  videoconference: 

1 . Students  alternate  across  sites  to  read  their  diary  entries. 

2.  Students  alternate  across  sites  to  perform  their  reenactments. 

3.  Students  alternate  across  sites  to  read  their  comparative  essays. 

Activities  back  in  the  classroom: 

1 . Share  what  they  remember  from  the  diaries,  reenactments,  and  essays. 

2.  Review  what  was  learned  about  the  reasons  for  and  events  of  the  battle  of  the  Alamo  and  the  different  perspectives  of 
Mexico  and  the  U.S. 

Approximately  a month  after  the  videoconference  in  which  students  explored  the  1836  battle  for  Texas,  the 
classes  videoconferenced  to  share  folktales  and  songs  with  each  other.  For  the  next  videoconference  students  created 
a school  mural  and  shared  pictures  of  the  murals  in  their  city  and  town  and  discussed  their  contents.  For  a final 
videoconference  they  exchanged  recipes  and  party  traditions  and  had  a shared  celebration.  The  Texans  had  a fiesta 
while  the  Mexicans  had  a party. 

The  students  participated  fully  in  the  above  activities  and  took  great  interest  in  each  other.  Through  the 
activities  students  tried  to  get  into  the  shoes  of  another  culture  and  acquired  curiosity  about  each  others’ 
perspectives.  Each  activity  led  to  a flood  of  student  questioning  regarding  the  life  of  the  distant  others.  Students  saw 
that  they  were  both  similar  and  different  from  each  other  in  many  ways.  Each  of  the  activities  initiated  by  the  project 
appeared  to  bring  the  children  together  for  greater  understanding  of  each  other’s  similarities  and  differences. 

Question  2:  What  similarities  and  differences  in  language  skills  and  pastimes  exist  between  the 
children  of  both  cultures? 

Five  of  the  41  Texan  students  knew  two  languages  while  the  rest  knew  only  English.  Second  languages 
were  Hindi,  Icelandic,  Spanish,  Chinese  and  Korean.  All  of  the  46  Mexican  students  were  at  least  bilingual  in 
Spanish  and  English  and  eight  of  the  Mexican  students  spoke  languages  other  than  Spanish  and  English.  Those 
languages  included  French,  Italian,  German,  Portuguese,  and  Hebrew.  As  was  expected,  the  two  groups  had  English 
in  common.  Although  the  Mexican  students  spoke  English  fluently,  the  American  students  had  difficulty  attending  to 
the  Mexicans’  speech  via  videoconference.  Understanding  the  strong  Spanish  accent  via  videoconference  technology 
required  great  concentration  on  the  part  of  the  Texan  fourth  graders.  In  addition,  that  most  Texans  could  only  speak 
or  read  in  English  proved  to  be  a significant  barrier.  For  instance,  we  had  to  have  the  Spanish  text  chapter  regarding 
the  Texas  Revolution  translated  so  that  the  students  could  study  the  Mexican  perspective. 

Learning  about  and  comparing  ancestry  was  important  to  the  project  because  it  helped  children  recognize 
that  we  are  multicultural  societies  and  are  interconnected  through  our  transience.  Students  across  groups  had,  of 
course,  different  ancestries,  but  were  pleased  to  learn  of  similarities  as  well.  Three  of  the  Mexican  children  were 
bom  outside  Mexico.  Two  were  bom  in  the  U.S.  including  one  who  was  bom  in  Houston.  Another  was  bom  in 
Caracas,  Venezuela.  All  other  Mexican  children  were  bom  in  Mexico.  Similarly,  only  two  of  the  Texan  children 
were  bom  outside  the  U.S.,  one  in  India  and  the  other  in  South  Korea.  Going  back  just  one  generation,  however,  the 
students  revealed  that  many  of  their  families  are  new  to  their  nations.  Seven  of  the  Mexican  children  had  parents  who 
were  bom  in  the  U.S.  and  nine  other  children  had  parents  bom  in  six  other  countries.  Ten  of  the  Texan  children  had 
parents  who  were  bom  outside  of  the  U.S.  Two  of  those  children’s  parents  were  bom  in  Mexico. 

Thorough  analysis  of  the  survey’s  contents  will  reveal  a great  deal  more  about  shared  and  disparate  interests 
among  the  children.  Preliminary  findings  inform  us  regarding  student’s  perceived  similarities  and  differences.  After 
students  met  to  share  their  collages,  their  discussion  led  to  their  drawing  the  following  conclusions:  ‘They  (the 
distant  class)  are  the  same  as  us!”  They  exhibited  the  same  taste  for  pizza  and  tacos,  admiration  of  hair  ribbons  and 
watches  for  the  girls,  high  regard  for  Michael  Jordan,  sports,  and  fast  cars,  for  the  boys,  and  appreciation  of  rock 
music  across  genders.  The  Texan  children  observed  that  the  Mexican  children  were  also  different  from  them  in  many 
ways.  They  all  spoke  at  least  two  languages;  their  names  all  sounded  different  from  the  Texans;  they  traveled  more 
than  the  Texan  children;  they  created  more  “decorative”  collages;  they  spoke  English  with  an  accent;  and  soccer  was 
generally  the  favorite  sport  (while  the  Texans  preferred  football  or  baseball).  All  but  one  Mexican  student  had  been 
to  the  U.S.,  while  only  two  Texans  had  been  to  Mexico. 

Question  3:  What  prevailing  impressions  did  Mexican  and  Texan  children  have  of  each  other  prior  to 
their  distance  learning  activities? 
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A preliminary  examination  of  the  poems  that  students  wrote  about  each  other's  countries  prior  to  meetings 
revealed  that  the  Mexican  children  had  enough  experience  of  U.S  culture  to  be  able  to  accurately  describe  the  culture 
in  poetry.  On  the  other  hand,  most  Texan  children  had  little  knowledge  of  Mexico  and  had  to  write  about  their  own 
culture,  create  an  imaginary  place,  or  refer  to  stereotypes.  During  the  poetry  workshop,  students  were  encouraged  to 
write  both  good  and  bad  things  about  each  other's  countries.  Most  importantly  they  were  to  be  honest  in  their 
imagery. 

Poetic  imagery  in  the  Mexican  student’s  poems  included  pervasive  institutions  such  as  Toys-R-Us  and 
shopping  malls;  defining  places  such  as  Disneyland  and  Las  Vegas;  prevalent  foods  such  as  ice  cream,  popcorn,  and 
hamburgers;  and  symbols  such  as  the  Statue  of  Liberty  and  the  Golden  Gate  Bridge.  Several  Mexicans  wrote  of  the 
clean  air  in  the  U.S.  that  contrasts  with  that  in  Mexico  City.  The  imagery  of  the  U.S  was  generally  positive  and 
accurate,  however,  some  Mexican  students  captured  both  negative  and  positive  aspects  of  the  U.S.  in  their  poems. 
One  student  referred  to  the  historic  connection  between  Mexico  and  the  U.S.  when  he  wrote,  “I  am  formed  by  old 
parts  of  Mexico.'’ 


I am  the  United  States  by  Nicholas  Aguilar 
I am  the  United  States, 

I am  a smelly  place, 

I am  the  taller  building, 

I am  scratching  the  sky, 

I am  teaching  children, 

I am  turning  them  bright, 

I am  a glorious  state, 

I am  playing  in  the  street, 

I am  breathing  air, 

I am  a tornado, 

I am  killing  them  all, 

I am  a little  polluted, 

I am  driving  all  around, 

I am  a car  blowing  his  hom, 

I am  going  down  on  a toboggan, 

I am  swimming  in  the  pool, 

I am  a noisy  place, 

I am  the  United  States, 

I am  Sylvester  Stalone, 

1 am  going  on  a bicycle. 


I am  the  United  States  by  Emiliano  Gordillo 
I am  a kid  playing  on  a patio 
I am  a house 

I am  a school  and  I see  kids  playing 

I am  green  grass  and  persons  step  on  me 

I am  water  flowing  through  the  Mississippi  River 

I am  a rock  that  blocks  water 

I am  while  sand  and  kids  play  with  me 

I am  the  United  States 

I am  an  acom  falling  from  a tree 

I am  light  and  I give  light  to  persons 

I am  the  flag  of  the  United  States 

I am  the  50  stars  of  the  flag  and  kids  sing  for  me 


I am  the  United  States  by  Jose  Lach 
I touch  land  and  I touch  all  the  happiness  and 
sadness. 

I am  a big  person  with  many  people  in  me  and  I can 
Not  move  and  if  I move  a disaster  comes  in  me. 

I am  the  biggest  in  America  and  I am  a person  with  a 

country  full  of  rivers 

I am  a country  with  many  people 

I am  a country  between  Mexico  and  Canada 

I am  full  of  big  shops,  and  you  can  come  and  visit  me 

And  it  will  make  me  happier  and  I’ll  make  you  happy 

I am  a very  old  place 

I am  formed  by  old  parts  of  Mexico 

If  you  come  I’ll  be  glad  to  make  you  play  and 
I’ll  be  glad  to  keep  you  safe. 

I am  very  clean 
1 am  not  full  of  pollution 


Poetic  imagery  in  the  Texan  students'  poems  included  language  from  the  poetry  writing  workshop  such  as 
graveyards,  beer  cans,  peaches,  and  churches.  Texans  seemed  to  draw  from  their  local  community  rather  than  from 
pictures  or  travels  in  Mexico  to  find  images  to  include  in  their  poems.  They  wrote  of  general  characteristics  such  as 
seasons  without  knowing  for  sure  that  they  were  describing  Mexico.  As  with  the  Mexican  writers,  the  Texans’ 
imagery  of  Mexico  was  generally  positive,  however  many  of  the  images  were  neither  unique  to  Mexico  or  even 
characteristic  of  Mexico.  Accurate  imagery  included  pinatas,  courtyards,  tortillas,  and  sombreros. 


I Am  Mexico  by  James  Knight 
1 am  a road 

I am  a peach  in  a person’s  hand 
I am  a sun  in  the  summer 
I am  snow  in  the  winter 
I am  leaves  in  the  fall 
I am  flowers  in  the  spring 
1 am  rocks  in  the  dirt 
I am  wood  in  the  weeds 

I Am  Mexico  by  Toura  Go 
I am  Mexico  with  tortillas  in  the  air. 

I am  a country  with  ghosts  around  me 
I am  summer  with  all  the  sun 
I am  the  moon  the  wind  the  sun 
I am  a store  with  Mexican  food 
I am  a country  with  hardly  any  rain 
I am  a country  with  lots  of  parades 
With  ghost  pinatas  and  Mexican  songs  we 
celebrate  a birthday 

My  sleep  is  filled  with  dreams  I never  had 
before 

I am  a country  called  Mexico 


I Am  Mexico  by  Cyndi  Hall 
1 am  Mexico 

I am  the  com  in  the  fields 

I am  the  drawbridge  between  rivers  and  dry 

land 

I’m  the  flowers  in  the  yard, 

I am  out  in  a blizzard 
I am  the  rain  in  a thunderstorm 
I am  the  waves  in  the  ocean 
I’m  the  fish  in  the  lake 
I’m  trees  that  grow  in  springtime 
I am  the  fire  in  the  fireplace 
I’m  the  home  on  someone’s  land. 

I’m  the  graves  in  the  courtyard 

I’m  the  guns  in  the  closet 

I am  chili  in  the  pan 

I am  horses  in  the  stalls 

I’m  the  church  in  the  grass 

I’m  the  clouds  in  the  sky 

I am  the  stars  shiny  way  up  high 

I’m  the  relationship  between  two  lovers 
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Educational  and  Scientific  Implications 

The  year  of  Cultural  Connections  between  Mexican  and  Texan  children  is  still  in  progress.  The  data 
described  above  have  been  collected.  However,  more  thorough  analyses  need  to  be  conducted  for  in-depth 
understanding  of  answers  to  the  research  questions.  Still,  the  story  of  this  case  of  international  team  teaching 
provides  a model  for  others  to  expand  their  learning  communities  beyond  national  borders.  We  have  learned  of  the 
importance  of  administrators  in  facilitating  the  processes  and  supporting  the  teachers  who  implement  distance 
learning  in  the  K-12  environment. 

As  we  continue  to  explore  the  research  questions  above  and  gain  insight  into  the  effects  of  distance  learning 
on  K-12  students,  we  will  also  ask,  “How  closely  do  students’  impressions  of  their  distant  partners  match  their  self- 
impressions?”  as  well  as,  “What  impressions  do  Mexican  and  Texan  children  have  of  each  other  after  a year  of 
distance  learning  together?”  By  comparing  perceptions  and  misconceptions  of  students  before  and  after  the  year  of 
experiences  we  will  gain  further  insight  into  the  effects.  We  will  also  compare  effects  of  multicultural  activities  that 
require  children  to  walk  in  each  other’s  shoes  vs.  those  that  involve  the  more  typical  approach  of  sharing  one’s  own 
culture. 

A Web  board  for  student  online  discussion  has  been  created  and  teachers  have  been  informed  for  months  of 
its  existence,  but  they  have  yet  to  have  their  students  use  the  space  for  collaborative  writing.  In  the  future  we  plan  to 
facilitate  integration  of  the  Web  board  as  weii  as  desktop  videoconferencing  to  support  student  collaboration.  By 
incorporating  a variety  of  distance  technologies,  students  will  have  greater  opportunity  to  gain  mutual  understanding. 
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Appendix  A 


Student  Survey 


1. 

Have  you  ever  traveled  to  another  country? 

8. 

What  kind(s)  of  music  do  you  like  to  listen  to? 

2. 

If  so  where? 

9. 

Do  you  create  art?  And  if  so,  describe  your  favorites 

3. 

How  many  languages  can  you  speak? 

(music,  drawing,  painting,  dancing,  sculpting,  or  writing). 

4. 

What  language(s)  do  you  speak? 

10. 

What  kinds  of  reading  do  you  like?  Books,  newspapers. 

5. 

What  language  do  you  speak  at  home  the  most? 

comic  books,  magazines,  etc. 

6. 

How  do  you  feel  about  studying  English? 

11. 

About  what  subjects  do  you  like  to  read? 

7. 

If  you  could  create  the  ideal  day,  how  would  you  spend 

12. 

How  do  you  know  right  from  wrong? 

it? 

13. 

If  there  is  time  and  you  would  like  to,  draw  a picture  here: 

Appendix  B 


THE  BATTLE  OF  COTTONWOOD  RIDGE 

Read  the  following  explanation  of  why  the  Battle  of  Cottonwood  Ridge  was  fought.  Pretend  you  are  from  The  Nation  or  from  The 
Country  and  write  a diary  entry  telling  why  you  believe  Cottonwood  Ridge  is  yours,  what  you  do  about  it  during  the  battle,  and  who  finally  wins 
Cottonwood  Ridge. 

There  once  war  a beautiful  place  called  Cottonwood  Ridge  where  Spaniards  set  up  a mission  whose  purposes  were 
to  (])  convert  native  Americans  to  Christianity,  (2)  educate  them,  and  (3)  establish  a Spanish  territory.  Cottonwood 
Ridge  served  as  a Spanish  mission  for  almost  100  years  until  a terrible  sickness  killed  all  of  the  people  living  there 
except  the  Native  Americans. 

People  who  were  ruled  by  Spain  lived  in  The  Nation  to  the  south  of  Cottonwood  Ridge  and  there  were  people  from  The  Country  to 
the  East  of  Cottonwood  Ridge.  People  from  The  Nation  and  people  from  The  Country  both  started  to  settle  the  land  around  Cottonwood  Ridge. 

Then  the  people  from  The  Nation  fought  a war  with  Spain  and  gained  independence  so  that  they  could  rule  themselves.  Part  of 
winning  the  war  meant  that  they  won  Cottonwood  Ridge  from  the  Spaniards.  They  banned  granting  land  to  people  from  The  Country. 

People  from  France  said  that  they  owned  Cottonwood  Ridge  and  they  sold  it  to  the  people  from  The  Country.  Now  the  people  from 
The  Nation  and  the  people  from  The  Country  both  believed  that  they  ruled  Cottonwood  Ridge.  They  fought  a battle  over  who  would  rule  the  land 
surrounding  Cottonwood  Ridge. 
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Purpose:  This  project  was  part  of  the  program  evaluation  of  the  Iowa  Distance  Education  Alliance,  a Star 
Schools  Project  funded  by  the  U.S.  Department  of  Education.  The  program  was  at  a point  where  it  was  necessary  to 
take  a step  back  and  examine  the  overall  widespread  usage  of  technology  in  Iowa’s  schools  to  serve  as  baseline  for 
future  activities  and  to  support  informed  decision  making. 

Design:  Descriptive  and  qualitative. 

Population:  Thirty  Iowa  high  schools  were  selected  to  represent  the  state.  Interviews  were  conducted  with 
students,  faculty,  and  administration  at  each  of  these  schools. 

Methods:  Results  from  interviews  with  students,  faculty,  and  administrators  were  transcribed  and  analyzed. 
Individual  case  studies  have  been  written  for  each  site  describing  the  setting,  current  level  of  technology  and 
technology  use,  perceived  impact  of  technology,  and  factors  contributing  to  the  current  state  of  technology  use  at  that 
site.  Additional  analysis  was  conducted  to  determine  relationships  among  the  various  variables  including  district 
size,  geographical  location,  reported  level  of  technology  use,  actual  technology  use,  perceived  impact,  and  critical 
factors. 

Findings:  The  primary  uses  of  technology  are  for  word  processing,  Internet  research,  and  vocational 
applications.  The  more  technologically  advanced  schools  are  incorporating  multimedia  production  into  the 
curriculum  as  well.  This  integration  of  technology  into  the  curriculum  is  occurring  despite  multiple  barriers,  the  most 
significant  of  which  are  lack  of  vision,  time,  money,  access,  training,  and  technology  support.  The  integration  of 
technology  in  Iowa’s  classrooms  has  triggered  some  important  changes.  Teachers  and  students  report  changes  in 
teaching  methods,  increased  student-directed  learning,  increased  access  to  information,  as  well  as  more  motivated 
learning  for  both  students  and  teachers. 


This  multi-site  case  study  was  conducted  as  part  of  the  program  evaluation  of  Iowa’s  Star  Schools  Project, 
Iowa  Distance  Education  Alliance.  Thirty  high  school  sites  were  selected  to  represent  the  state  of  Iowa  based  on: 


Purpose 


1.  district  enrollment, 

2.  rural  or  urban  setting, 

3.  technology  use,  and 

4.  geographical  location. 
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Data  collection  incorporated  document  review,  site-visit  observations,  one-on-one  interviews,  and  focus 
group  interviews.  In  addition,  survey  data  collected  by  the  Iowa  Department  of  Education  provided  demographic  and 
technology  data  for  each  site  selected.  This  study  provided  both  qualitative  and  quantitative  baseline  data  about  the 
status  of  technology  use  in  education  in  Iowa.  Research  questions  included: 

1.  How  is  technology  making  a difference  in  the  learning  process  for  Iowa  high  school  students? 

2.  How  does  technology  change  what  and  how  teachers  teach? 

In  general,  what  is  the  current  state  of  technology  and  technology  use  in  Iowa  high  schools? 

Rationale 

Statewide  research  on  technology  integration  in  Iowa  schools  has  focused  on  quantitative  data.  Specifically, 
the  Iowa  Department  of  Education  technology  survey  has  requested  information  on  the  number  of  computers  in  the 
school,  the  number  of  students  per  computer,  the  number  of  schools  with  access  to  the  Internet  and  other  numerical 
data.  This  baseline  data  was  intended  to  “describe  the  status  of  Iowa  public  school  technology,  to  reflect  needs,  and 
to  facilitate  the  school  improvement  process”  (EDOE,  1997). 

While  statistics  are  helpful,  they  don’t  seem  to  hold  up  to  closer  scrutiny.  How  schools  interpreted  the 
questions  varied  widely  among  districts.  In  computing  the  number  of  students  per  computer,  many  districts  included 
computers  that  are  dedicated  to  administrative  uses  and  inaccessible  to  students.  In  addition,  older  model  computers 
(only  capable  of  word  processing)  were  counted  the  same  as  more  powerful  computers  capable  of  full  multimedia 
computing.  Schools  reported  Internet  access  even  if  the  only  access  they  had  was  one  line  in  the  administrator’s 
office  or  the  media  center. 

Technology  integration  is  more  than  installing  hardware  and  software.  In  order  to  understand  the  process  of 
change  involved  with  technology  integration  it  is  imperative  to  speak  with  the  people  involved.  The  baseline  study 
focused  on  the  people  involved  in  integrating  technology  in  the  schools  rather  than  counting  hardware  and  software. 
To  better  understand  the  change  process  as  it  relates  to  technology  integration,  it  is  important  to  know  what 
motivates  districts  to  integrate  technology  into  the  curriculum,  how  problems  are  addressed,  and  how  technology  is 
changing  teaching  and  learning.  This  information  will  assist  policy  makers  to  determine  where  additional  support  is 
needed  as  the  technological  landscape  in  our  schools  continues  to  change. 

Relevant  Literature  Review 

Despite  the  potential  many  believed  technology  had  for  revolutionizing  education,  education  has  changed 
little  in  the  past  decades  (David,  1994).  While  access  to  technology  in  schools  continues  to  increase,  the  overall 
change  in  teaching  and  learning  has  been  minimal  (Zappone,  1991).  The  International  Society  for  Technology  in 
Education  (ISTE)  identified  the  need  for  teachers  to  model  computer  use  in  the  classroom  and  to  naturally  integrate 
technology  in  the  curriculum  in  relevant  ways  that  prepare  students  to  face  the  challenges  of  the  next  century  (Friske, 
et.  al.,  1995).  However,  few  schools  are  succeeding  in  helping  teachers  and  students  actually  infuse  computers  into 
their  daily  lives.  (OTA,  1995). 

Change  theory  provided  a framework  for  examining  the  process  that  occurs  when  districts  attempt  to 
integrate  technology  into  the  curriculum.  Because  of  its  focus  on  the  individuals  involved  in  the  change  process, 
Hall’s  Concerns  Based  Adoption  Model  (CBAM)  was  particularly  useful  (Hall,  1974b).  CBAM  provided  a structure 
for  examining  the  very  complex  process  that  occurs  when  educational  institutions  become  involved  in  adopting 
innovations. 

CBAM  views  the  teacher  as  the  focal  point  in  school  improvement  efforts,  yet  acknowledges  significant 
social  and  organizational  influences  as  well.  The  model  focuses  on  two  facets  of  the  individual’s  developmental 
growth  in  relation  to  the  innovation.  These  components  provide  a reference  for  interpreting  an  individual’s  level  of 
concern  and  level  of  use  in  relationship  to  technology  integration. 

Sample 

In  order  to  capture  the  true  reality  of  technology  in  Iowa’s  high  schools,  30  schools  were  selected  to 
represent  the  state.  In  addition  to  requiring  two  schools  from  each  of  the  15  Regions  in  the  state,  we  used  three 
selection  criteria:  population  density,  enrollment  and  current  technology  level. 

The  two  charts  shown  below  compare  the  schools  we  visited  to  all  public  school  districts  based  on 
population  density  and  enrollment.  We  tried  to  reflect  the  state  on  these  statistics.  Population  density  (Chart  1) 
ranged  from  10.5  individuals  per  square  mile  to  a high  of  2388.5  people  per  square  mile.  The  number  of  students 
(Chart  2)  in  the  district  ranged  from  a low  of  202  to  a high  of  32,033. 

The  results  of  the  State  Department  of  Education  Technology  survey  were  used  to  determine  current  levels 
of  technology.  Across  the  state  the  average  number  of  students  per  computers  was  about  six.  The  15  high  technology 
schools  visited  had  fewer  than  four  students  per  computer.  In  contrast,  the  15  low  technology  schools  reported  an 
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average  of  over  14  students  per  computer.  All  schools  said  they  had  Internet  access  and  sixty  percent  were  connected 
to  the  Iowa  Communications  Network. 


Chan  1 : Comparison  of  Statewide  Districts  and  Visited  Districts  by  Population  Density 


Chart  2.  Comparison  of  Statewide  districts  and  Visited  Schools  by  District  Enrollment 
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We  covered  every  comer  and  region  of  the  state  in  our  desire  to  understand  technology  integration  in  Iowa. 
We  traveled  to  schools  in  small  town  Iowa,  consolidated  schools  located  in  the  middle  of  farmland,  and  major  urban 
area  schools.  The  level  of  technology  we  found  was  as  diverse  as  the  settings.  Frequently,  our  assumptions  about 
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where  we  thought  we  would  find  high  levels  of  technology  integration  proved  to  be  wrong.  Some  seemingly  isolated 
schools  were  surprisingly  technology  rich.  We  visited  schools  where  technology  was  not  recognized  as  a high 
priority.  We  also  visited  schools  where  both  teachers  and  administrators  have  identified  the  need  for  technology 
integration  and  actively  planned  for  effective  integration  of  technology  into  the  school  curriculum.  And,  as  was  to  be 
expected,  we  found  many  schools  in  varying  degrees  between  the  two  extremes. 

Limitations  of  the  Study 

Three  delimiting  factors  of  the  study  were  identified:  technology  level,  teacher/student  selection,  and 
researcher  biases.  Technology  level  was  used  as  a site  selection  criteria.  However,  pupil  to  computer  ratios  varied 
considerably  around  the  state,  between  AEAs,  and  within  each  AEA.  The  selection  of  a high  and  low  technology 
school  within  each  AEA  provided  more  of  a continuum  representing  a range  of  technology  levels  instead  of  a 
statewide  representation  of  extremes. 

Researchers  had  little  or  no  control  over  the  selection  of  students  and  teachers  for  the  focus  group  sessions. 
While  a random  group  was  requested,  at  some  sites  it  appeared  that  teachers  particularly  had  been  selected  for  their 
technology  expertise.  It  is  suspected  that  our  sample  of  teachers  may  be  slightly  skewed  as  a result  of  this.  Student 
selection  appeared  to  be  more  random. 

The  third  delimiting  factor  resulted  from  the  number  of  researchers  required  for  the  data  collection  phase  of 
this  study.  While  all  researchers  were  trained  and  provided  with  guidelines  and  questions,  natural  biases  in  the  way 
the  questions  were  asked  and  the  prompts  used  may  have  affected  the  responses  of  individuals  participating  in  the 
focus  groups  and  one-on-one  interviews. 

Data  Collection 

Because  of  the  political  interest  in  this  study,  a large  number  of  sites  were  selected  to  allow  generalizability. 
Thirty  high  schools  were  selected  to  represent  a cross-section  of  the  state.  All  Area  Education  Agencies  (AEAs)  were 
represented.  In  addition,  sites  were  selected  to  reflect  state  statistics  on  enrollment,  population  density,  and 
technology. 

A site  visit  was  arranged  with  each  site  to  include  one-on-one  interviews  with  the  principal  and  the 
technology  coordinator,  focus  group  interviews  with  a group  of  teachers  and  with  a group  of  students,  and  a tour  of 
the  facility.  All  research  teams  were  trained  and  provided  with  interview  guides  and  observation  checklists.  One 
researcher  was  responsible  for  recording  all  interviews  but  to  insure  accuracy  all  interviews  were  taped  except  in  rare 
instances  where  permission  was  denied. 

Data  collection  began  during  the  third  week  of  March  1998,  and  continued  through  the  middle  of  May. 

Each  of  the  thirty  selected  high  schools  was  visited  for  the  majority  of  a school  day  by  two  researchers.  The  typical 
visit  began  with  a one-hour  interview  with  the  high  school  principal,  followed  by  a tour  of  the  school.  The  tour  was 
usually  led  by  the  technology  coordinator,  a teacher  highly  involved  in  technology  at  the  school,  or  the  principal.  The 
researchers  usually  spent  the  lunch  hour  in  the  faculty  dining  area,  discussing  technology  casually  with  teachers.  The 
focus  group  with  students  was  often  held  right  after  lunch,  typically  with  a random  group  of  students  who  were 
assigned  to  study  hall  for  that  period.  The  teacher  focus  group  was  typically  held  right  after  school. 

The  data  gathered  at  each  school  was  used  to  write  a case  study  for  each  site  visited.  Primarily  the  lead 
researcher  for  each  site  wrote  these  case  studies,  but  the  assistant  researcher  reviewed  the  case  study  draft  and 
provided  input  and  clarification.  This  member  checking  process  allowed  for  review,  verification  and  comment. 
Conclusions  were  revised  and  clarified  based  on  recommendations  from  the  member-checking  process. 

Next,  the  text  of  the  focus  group  sessions  and  principal  interviews  was  formatted  for  use  with  a computer 
software  program  for  qualitative  research.  The  Non-numerical  Unstructured  Data  Indexing  Searching  and  Theorizing 
(NUD*IST)  software  allows  for  efficient  management  of  the  large  amount  of  data  collected  for  the  project,  detailed 
exploration  of  the  data,  and  makes  it  easier  to  see  patterns  in  the  data. 

Two  researchers  coded  the  text  of  the  focus  group  sessions  and  interviews.  Each  researcher  coded  data  into 
a core  group  of  agreed  upon  categories,  and  each  researcher  was  allowed  to  create  new  categories  as  needed.  After 
all  documents  had  been  coded,  the  researchers  discussed  and  justified  additions.  Some  coding  categories  were 
combined  at  this  point. 

When  the  coding  phase  of  the  research  was  completed,  researchers  checked  inter-coder  reliability  to  look 
for  potential  coding  problems.  Four  key  categories  with  subcategories  were  chosen  to  test  reliability.  There  was 
complete  agreement  (100%)  in  three  of  the  four  categories.  The  category  called  “vision”  had  a very  low  reliability  - 
25%.  The  original  four  subcategories  were  not  discrete,  and  therefore  it  was  difficult  to  place  some  text  units  in  only 
one  category.  The  four  subcategories  were  then  collapsed  into  two  categories,  making  them  discrete. 

The  focus  group  data  was  then  triangulated  with  other  data  collected  at  the  site:  the  principal  interview,  the 
focus  group  discussion  with  students  at  the  school,  the  technology  coordinator  interview,  observational  data,  and  the 
district  technology  plan.  Triangulation  helped  to  pinpoint  the  accuracy  of  conclusions  through  the  use  of  several 
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sources  of  data.  It  also  helped  prevent  the  researcher  from  accepting  the  validity  of  his  or  her  initial  impressions 
without  further  substantiation  and  served  to  control  for  researcher  biases. 

Results 

Analysis  of  data  indicated  a great  diversity  of  technology  integration  in  Iowa’s  high  schools.  Original 
thoughts  about  a relationship  between  size  of  district  and  level  of  technology  do  not  appear  to  be  supported  by  the 
data.  In  addition,  data  seems  to  conflict  with  previous  conceptions  related  to  the  level  of  technology  in  rural  districts. 
Data  also  seems  to  be  providing  a clearer  picture  of  technology  than  was  available  with  previous  quantitative  data. 

A look  at  the  current  state  of  technology  in  Iowa  High  Schools  indicated  four  most  common  uses  of 
technology  by  both  teachers  and  students.  This  included  word  processing,  Internet,  vocational  applications,  and 
multimedia  uses.  The  vast  majority  (over  80%)  of  the  teachers  and  students  in  our  sample  reported  having  a 
computer  at  home  and  approximately  50%  indicated  they  had  Internet  access  at  home. 

When  teachers  were  asked  to  define  integration  of  technology,  the  majority  identified  technology  as  a tool. 
Many  indicated  that  integrating  technology  allowed  them  to  vary  the  instructional  approach  and  to  provide 
educational  experiences  that  would  be  difficult  to  do  otherwise.  Many  saw  technology  integration  as  the  way  to 
connect  schoolwork  to  the  “real”  world. 

There  was  a difference  in  response  between  teachers  at  high  technology  schools  and  teachers  at  low 
technology  schools.  Teachers  at  low  technology  schools  seemed  to  focus  on  why  they  couldn’t  integrate  technology. 
The  lack  of  access  seemed  to  cloud  their  vision  of  what  might  be  possible. 

Teachers  identified  several  factors  that  motivated  them  to  integrate  technology.  One  major  factor  was 
increased  enthusiasm  for  learning.  Teachers  indicated  their  enthusiasm  increased  in  addition  to  their  students’ 
enthusiasm  for  learning  when  technology  was  used  as  part  of  the  curriculum.  This  encouraged  both  teachers  and 
students  to  get  excited  about  the  content  area  in  new  ways.  Teachers  also  indicated  that  they  used  technology  because 
it  helped  them  to  prepare  students  for  the  future.  They  saw  student  preparation  for  entering  either  the  workforce  or 
college  as  a primary  motivating  factor  for  using  technology.  Other  motivating  factors  included  administrative  support 
or  directive,  and  the  influence  of  colleagues. 

Again,  there  appeared  to  be  a difference  between  high  and  low  technology  schools.  Teachers  at  low 
technology  schools  tended  to  avoid  identify  motivating  factors  and  instead  point  out  why  they  couldn’t  use 
technology. 

It  became  evident  that  there  were  several  barriers  to  technology  integration.  Barriers  included  inadequate 
training,  inadequate  access,  lack  of  time,  and  lack  of  a common  vision  for  technology  integration.  Of  these,  vision 
seemed  to  be  the  most  critical. 

Where  technology  was  being  successfully  integrated,  there  was  a vision  holder  who  could  clearly  identify 
expectations  of  technology  use.  In  some  schools  the  vision  holder  was  an  administrator,  either  the  superintendent  or 
principal,  in  others  it  was  a group  of  teachers. 

What  we  want  to  do  is  make  the  technology  so  that  when  you  come  into  a classroom  you  don’t  come  out  thinking 
“what  a wonderful  technology  we  have  in  the  classroom”,  you  come  out  thinking  about  the  wonderful  student 
learning  that  is  going  on.  Whether  it  is  a microscope,  scanner,  or  computer,  we  want  that  technology  to  assist  all 
our  students  so  that  they  can  leam.  It’s  a big  shift  in  the  way  our  teachers  will  be  presenting  in  format  ion,... There 
are  still  a lot  of  good  things  happening  in  terms  of  hands  on  learning,  but  we  want  technology  to  come  in  where  it’s 
very  pertinent  and  very  relevant. 

Principal 

Where  technology  was  being  integrated,  teachers  reported  several  changes  occurring.  First  of  all  was  the 
role  of  the  teacher.  Classrooms  become  more  student  centered  and  the  teachers  become  facilitators  of  learning.  Use 
of  lecture  is  down  and  students  are  given  more  responsibility  for  their  own  learning. 

Not  only  did  the  role  of  the  teacher  change  but  the  role  of  the  student  also  changed.  Students  were  more 
motivated.  They  were  going  out  and  getting  information  rather  than  waiting  for  a teacher  to  provide  the  information. 
Technology  also  allowed  teachers  to  individualize  lessons  and  meet  the  needs  of  a variety  of  students. 

Other  important  changes  identified  by  teachers  included  additional  information  sources,  increased 
efficiency  and  increased  teacher  collaboration/cooperation. 

In  summary,  we  found  that  the  majority  of  teachers  view  technology  as  a tool.  They  are  motivated  to  use 
technology  because  of  their  own  desire  to  leam  and  grow  professionally  and  by  their  desire  to  provide  this  same 
opportunity  for  their  students.  While  there  are  many  barriers  to  successful  integration  of  technology,  schools  with  a 
vision  are  overcoming  these  barriers  and  seeing  important  changes  occurring  in  the  classroom. 

Significance 

Two  aspects  of  this  study  make  significant  contributions  to  the  field  of  program  evaluation  and  technology 
integration.  The  methodology  used  in  this  study  serves  as  a model  for  others  conducting  needs  assessments  or  impact 
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studies  related  to  program  evaluation.  In  addition,  the  results  of  this  study  provided  insight  into  the  change  process 
related  to  technology  integration.  It  allows  stakeholders  to  make  informed  decisions  and  allocate  funds  and  programs 
so  technology  can  make  the  most  impact  on  teaching  and  learning  and  overall  school  improvement. 

Recommendations  made  to  stake  holders  identified  three  areas  that  need  to  be  addressed  to  assist  schools  in 
making  a successful  transition  to  full  integration  of  technology.  These  areas  included  training,  access,  and  technology 
vi  sion/planning. 
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Introduction 

The  purpose  of  this  paper  is  examine  how  discourse  analysis  can  be  used  as  a methodology  for 
understanding  the  mental  models  that  influence  how  people  navigate  the  Internet.  Discourse  analysis  provides  a 
methodology  with  which  to  articulate  the  relationships  between  language,  social  context,  and  the  cognitive  processes 
that  underlie  discourse  perception  and  production.  This  paper  describes  the  results  of  a pilot  study  that  asked 
participants  about  how  they  navigate  the  Internet,  as  well  as,  how  they  conceptualize  space  and  time  while  using  the 
World  Wide  Web.  By  providing  a clearer  sense  of  how  people  construct  their  experience  while  on  the  Internet,  this 
study  begins  to  present  techniques  for  analyzing  end-users  that  help  to  clarify  more  culturally  relevant  technology- 
based  pedagogies.  The  goal  of  this  paper  is  to  discern  strategies  for  studying  people  to  help  design  educational 
media,  like  educational  web  sites,  that  not  only  take  into  consideration  how  educational  media  is  structured,  but  that 
also  consider  the  guiding  mental  models  that  influence  people's  learning  on  the  Internet.  In  meeting  this  goal  this 
paper  argues  that,  "It’s  not  just  what  you  do  while  learning  on  the  Internet,  or  even  just  how  you  are  doing  it,  it’s 
always  who  you  are  and  what  you're  doing  when  you  are  learning.” 

“Discourses”  have  come  to  mean  more  than  language  to  include  such  categories  as  visual  representations,  as 
well  as  texts  (Fairclough,  1992),  including  World  Wide  Web  content  that  combine  words  and  images.  For  the 
purposes  of  this  study,  discourses  analysis  will  mean  studying  how  language  is  embedded  in  social  practice  through 
which  diverse  social  entities  are  formed  and  transformed  as  the  result  of  history  and  practice.  In  this  sense, 
discourses  are  always  social  and  the  result  of  histories.  All  of  the  discourses  that  an  individual  participates  in  are 
based  upon  individual  identities  and  social  affiliations  that  can  be  incompatible  and  produce  conflicts  that  are  often 
confronted  based  upon  individual  social  positions.  Discourse  analysis  provides  tools  with  which  to  describe  how  the 
complexities  of  multiple  and  conflicting  social  identities  influence  how  individuals  navigate  and  understand 
electronic  information  spaces.  This  study  will  use  discourse  analysis  to  begin  to  articulate  the  mental  pictures  that 
influence  people's  perceptions.  I will  call  these  mental  pictures  cultural  models. 

Cultural  models  have  been  developed  by  Gee  (1996)  and  Strauss  and  Quinn  (1997)  to  study  mental  models 
in  social  contexts.  While  Gee's  approach  to  cultural  models  describes  how  people  conceptualize  literacy  and  the 
social  and  political  implications  of  these  perceptions,  Strauss  and  Quinn’s  approach  tries  to  integrate  a cognitive 
approach  to  understanding  opinion  formation  by  showing  how  individuals  negotiate  attitudes  based  upon  internal 
schemas  that  are  influenced  by  social  and  cultural  contexts. 

The  Internet  provides  a unique  context  for  educational  research.  Not  only  is  it  increasingly  occupying  the 
public  imagination  as  an  important  medium  for  learning,  it  is  also  being  positioned  by  large  telecommunication, 
computer  industry,  and  entertainment  corporate  interests  as  a medium  for  people  to  carry  out  the  daily  functions  of 
middle  class  life  such  as  shopping,  banking,  entertainment,  etc..  Because  the  Internet  can  now  be  accessed  from  a 
variety  of  locations  including  home,  school,  work,  the  library,  etc.  and  because  of  the  decentralized  nature  of  Internet 
content,  the  World  Wide  Web  allows  for  different  conceptions  of  time  and  space  compared  to  other  educational 
media  like  video  and  CD-ROMs.  Depending  upon  how  one  has  access  to  the  Internet,  it  is  open  24  hours  a day,  7 
days  a week  in  any  place  in  the  world  where  access  exists.  One’s  sense  of  time  and  space,  experienced  through 
cultural  models  about  time  and  space,  can  become  compressed  or  expanded  depending  on  how  one  approaches  this 
medium.  One’s  cultural  models  define  much  of  the  Internet  learning  experience.  This  study  begins  to  see  how  people 
are  using  the  Internet  in  relation  to  how  they  conceptualize  themselves,  Internet  content,  and  time  and  space  while 
navigating  the  World  Wide  Web,  as  well  as,  if  it  is  possible  to  discern  the  cultural  models  that  people  bring  to  these 
areas  of  interest. 

Four  people  were  interviewed  for  this  study  and  their  answers  were  audio  taped  and  transcribed.  Each 
interview  took  approximately  20-30  minutes.  Participants  were  asked  questions  about  how  they  used  the  Internet, 
what  they  liked  and  disliked  about  particular  web  sites  that  they  frequent,  and  about  their  experiences  in  relation  to 
the  way  that  they  conceptualize  time  and  space  while  using  the  web.  Specific  questions  asked  of  each  participant 
include: 


1 . How  long  have  you  been  using  computers? 

2.  How  long  have  you  been  using  the  World  Wide  Web? 

3.  How  often  do  you  use  the  web  each  week? 

4.  How  long  do  you  use  the  web  each  day? 

5.  Tell  me  about  a site  that  you  particularly  like. 

6.  Tell  me  about  a site  that  you  particularly  dislike. 

7.  For  what  kinds  of  things  do  you  use  the  web? 

8.  What  is  your  perception  of  time  when  you  use  the  Internet? 

Follow-up  questions  were  then  asked  to  clarify  their  answers  to  the  above  questions. 

Cultural  Models 

Gee  (1996)  calls  cultural  models,  “pictures  of  simplified  worlds  in  which  prototypical  events  unfold.’’  He 
notes  that  people  are  often  unaware  of  their  cultural  models  and  therefore  cultural  models  appear  to  be  “natural”  and 
inevitable  even  though  cultural  models  vary  across  social  groups  and  change  over  time.  According  to  Gee,  ‘They 
allow  us  to  function  in  the  world  with  ease,  but  at  the  price  of  stereotypes  and  routinized  thought  and  perception.” 
Cultural  models  form  the  basis  of  choices  and  guesses  about  meaning  within  particular  communities  --  they  always 
include  a conception  of  what  is  acceptable  and  unacceptable  to  do  within  that  cultural  model.  Gee  states,  “It’s  not 
just  what  you  say,  or  even  just  how  you  say  it,  it’s  always  who  you  are  and  what  you’re  doing  when  you  say  it.” 

Strauss  & Quinn’s  work  has  shown  how  people’s  cultural  models  interact  with  each  other  and  how  they 
interact  within  various  contexts.  Cultural  models  vary  in  both  content  and  form.  The  process  of  cultural  model 
interaction  can  take  the  form  of  integration;  as  people  modify  their  cultural  models  to  make  them  more  consistent 
with  each  other.  They  also  observe  that  people  are  capable  of  possessing  inconsistent  cultural  models  without 
experiencing  a sense  of  dissonance.  They  call  the  process  of  maintaining  two  inconsistent  models 
compartmentalization.  According  to  Strauss  and  Quinn  belief  systems  are  partly  integrated  and  partly 
compartmentalized.  When  looking  at  peoples  cultural  models,  Strauss  and  Quinn  observed  three  things:  1)  the 
content  (i.e.  the  ideas  expressed)  of  people’s  talk,  2)  the  voice  (i.e.  characteristic  mode  of  expression)  they  use  to 
express  the  content,  and  3)  the  temporal  continuity  and  discontinuity  (i.e.  expressed  in  connected  or  disconnected 
discourse  contexts)  in  the  expression  of  the  content.  The  analysis  of  the  interview  transcripts  focused  on  these  three 
areas  of  observation,  as  well  as,  the  social  language(s)  that  participants  used  to  answer  the  questions. 

A social  language  conveys  a socially  situated  identity  that  communicates  affiliations  based  on  profession, 
ethnicity,  race,  gender,  nation,  etc.  (i.e.  social  identities),  as  well  as,  communicates  status  within  and  between  those 
affiliations.  Individual  often  switch  between  social  languages  depending  on  who  they  are  communicating  with  and 
how  they  want  to  communicate  themselves  to  the  other  person.  Looking  at  a person’s  social  language  reveals  who 
they  think  themselves  to  be  within  a communication  and  who  they  think  their  audience  to  be. 

Participant  Profiles 

The  four  participants  represent  a convenience  sample  of  Education  graduate  students  at  a large  Mid-west 
research  university.  They  have  been  using  the  Internet  for  at  least  2 years.  Pseudonyms  are  used  in  this  paper  to 
assure  participant  confidentiality.  Participants  were  asked  to  be  a part  of  a study  that  is  interested  in  looking  at  the 
ways  that  people  use  the  Internet.  In  order  to  help  avoid  sensitizing  the  participants  to  the  questions,  participants 
were  not  given  specifics  about  the  study  until  after  the  interviews  when  the  study  was  explained  to  them  in  more 
detail. 

The  profiles  of  the  four  participants  include: 

Ron  -Ron  is  a European  American  in  his  early  forties.  He  served  in  the  military  during  his  twenties  where 
he  worked  in  electronics.  After  his  tour,  he  worked  as  a Park  Ranger  and  educator  for  the  US  Park  Service,  before 
becoming  a school  district  technology  coordinator.  In  the  past  year  he  has  returned  to  school  to  pursue  a Ph.D.  in 
Educational  Technology.  Ron  has  a great  deal  of  experience  using  computers  and  helping  others  use  computers  and 
uses  the  Internet  two  to  three  hours  a day  to  look  at  technology  news  and  current  events. 

Patricia  - Patricia  is  a Mexican  American  in  her  mid-twenties.  She  is  bilingual  and  grew  up  in  Austin, 
Texas  where  she  visits  frequently  to  see  her  family.  She  earned  her  Bachelors  degree  in  Communication  Arts  and  is 
currently  employed  in  the  production  department  of  a public  television  station.  She  is  also  presently  working  on  her 
Master’s  degree  in  Educational  Technology  and  eventually  hopes  to  design  new  media,  like  CD-ROMs,  for  children. 
She  is  particularly  interested  in  creating  new  media  that  are  geared  towards  multilingual  children. 

Laura  - Laura  is  a European  American  in  her  late  twenties.  She  earned  an  undergraduate  degree  in 
Secondary  Education,  German,  and  Political  Science.  Currently  she  is  employed  as  a librarian  in  the  university’s 
Instructional  Media  Center.  She  is  working  on  her  Master’s  degree  in  a Library  graduate  school  where  she  is 
studying  to  become  a library  media  specialist.  As  a library  media  specialist  she  hopes  to  help  teachers  integrate  the 
Internet  into  their  curriculum.  She  considers  her  experience  using  the  Internet  and  computers  as  intermediate. 
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Jan  - Jan  is  a European  American  in  her  late  thirties.  She  earned  her  bachelors  degree  in  art.  She  has  spent 
most  of  her  working  life  in  advertising  as  a graphic  design  consultant  in  marketing  and  advertising.  Currently  she  is 
teaching  in  the  Journalism  department  at  the  university  in  which  this  study  was  conducted  where  she  is  also  working 
on  her  Ph.D.  Of  the  three  participants  she  is  the  least  experienced  with  computers.  During  the  time  of  the  study,  she 
had  just  purchased  a new  computer  and  was  looking  forward  to  spending  more  time  learning  new  software  and 
spending  time  on  the  Internet. 

All  of  the  participants  had  frequent  access  to  the  Internet  either  through  the  university,  work,  or  at  home  and 
often  through  various  combinations  of  the  three.  The  participant’s  experience  with  computers  and  the  Internet  is 
summarized  in  table  I . 

Table  I 


Participant 

Computer  Experience 

Internet  Experience 

Internet  Use 

Ron 

Advanced 

4 years 

Daily 

1 0- 1 2 hrs  / week 

Patricia 

Low  to  Moderate 

1 -2  years 

Not  Daily 
3 hrs/ week 

Laura 

Moderate 

3 years 

Daily 

2-3  hours/week 

Jan 

Low 

1 year 

Not  Daily  (2-3  times/week) 
3-4  hours/week 

Data  Analysis 


Since  the  purpose  of  the  study  was  to  begin  to  find  the  participants  cultural  models  of  Internet  usage  and 
conceptions  of  time  and  space  when  using  the  Internet,  the  analysis  was  divided  into  sections  of  usage,  time  and 
space.  The  analysis  will  show  schemas  present  for  each  of  these  categories,  as  well  as,  how  at  times  these  categories 
overlap,  reference  each  other,  and  occasionally  contradict  each  other.  For  the  purpose  of  this  paper  schemas  will 
represent  the  major  themes  that  emerged  form  the  interviews.  Schemas  will  act  as  data  organizers,  as  well  as, 
indicators  of  cultural  models.  The  data  suggested  that  the  participants  had  not  spent  a great  deal  of  time  reflecting  on 
the  ways  that  they  use  the  Internet;  therefore  they  rarely  consciously  went  on  a meta-cognitive  level  in  their,  mostly 
short,  answers.  This  seemed  to  be  the  first  time  that  they  had  thought  about  these  issues. 

It  appeared  as  if  the  more  time  that  the  participants  had  spent  using  the  Internet,  the  more  that  they  felt 
capable  of  analyzing  their  Internet  usage.  This  relationship  was  generally  also  true  for  their  confidence  in  critiquing 
Internet  content.  Patricia,  for  example,  gave  “contextualizing  signals”9  (i.e.  a speaker’s  signals  about  what  they  take 
the  context  to  be  and  what  sort  of  person  s/he  takes  the  listener  to  be)  that  indicated  that  she  was  not  confident  in  her 
answers.  She  often  criticized  her  answers  and  devalued  her  opinions.  The  following  sample  from  Patricia’s  interview, 
occurred  just  before  she  talked  about  her  favorite  web  site: 

wool 

you  can  go  to  the. 
taa 

wool,  it’s  interesting 

cause  usually  you  kinda 

I’m  trying  to  think 

cuz  I haven’t  been  on  it  in  a while 

but 

Usage 

Most  of  the  data  collected  in  the  interviews  relates  to  how  the  participants  use  the  Internet  in  both  their 
professional  and  personal  lives.  The  major  schema  that  emerged  from  all  the  participants  was  the  Internet  as 
Information  Resources  schema.  This  schema  constructs  the  Internet  as  a place  to  retrieve  information  that  benefits 
one’s  school  or  professional  work,  as  well  as,  a place  to  acquire  information  for  one’s  personal  use  like  travel 
information.  As  will  be  related  below,  this  schema  perhaps  best  fits  their  social  identities  as  graduate  students. 

Within  this  schema  there  seemed  to  be  two  subtle  distinctions,  one  developed  by  the  participants  who  had  used  the 
web  in  a business  context,  and  the  other  who  had  only  used  the  web  in  school.  The  participants  who  had  used  the 
web  as  an  information  resource  in  business  (Ron  & Jan)  seemed  to  be  much  more  focused  and  deliberate  in  the  way 
they  used  the  web  than  those  who  had  just  used  it  predominantly  at  the  University  (Patricia  & Laura). 

The  following  quotes  help  to  show  how  the  Internet  as  Information  Resources  schema  gets  constructed: 
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Patricia: 

3.  Most  of  the  things  that  I use  it 
umm...for  information 
things  mainly  related  to  school 
I really  don’t  use  it  a whole  lot  as  personal 
and  I want  to 
but  I just  for  some  reason 
I just  don’t 

Interviewer:  How  often  do  you  think  you  use  it  a week,  like  how  many  hours  would  you  approximate? 


Patricia: 

1.  Hours  a week? 

Not  very  much. 

I’d  probably  have  to  say  3 
I mean,  you  know 

Sometimes  more  like  when  I’m  trying  to  find  a lot  of  things 

like  where  I’m  trying  to  find  clip  art 

or  I’m.  I have  to  find  information 

then  I probably  spend  a lot  more 

but  I’ve  never  actually  calculated 


2.  I mean  I can  spend  2-3  hours  a day  on  it 

and  that’s  finding 

trying  to  find  information  that  I need 
but  that  doesn’t  come  a lot 
that  doesn’t 
I don’t  usually  try  to 
because  I usually  do  it  a little 


Patricia  sees  herself  as  different  from  the  “norm’'  of  Internet  users  because  she  doesn’t  feel  as  if  she  uses  it 
for  personal  reasons,  calling  herself  an  “oddball”  because  of  this.  Later  in  the  Interview  she  does  state  a personal  use 
of  the  Internet  by  observing  that  she  uses  it  to  look  up  airline  fares.  Even  this  practice  has  a utilitarian  component 
since  she  mostly  flies  back  to  Austin  to  be  with  her  family.  “I  go  to  Austin  a lot”.  Living  in  two  cultures,  spending 
time  in  her  family’s  culture  seems  to  be  important  to  her.  Nevertheless,  this  falls  firmly  into  the  Internet  as 
Resource  schema. 

Like  Jan,  Patricia  said  she  does  not  use  it  for  “pleasure”  when  she  is  in  school,  “When  school  starts  I do 
everything  in  relation  to  school.”  It  is  interesting,  however,  that  when  asked  to  name  a site  that  she  particularly  likes 
it  is  Martha  Stewart’s  site,  a place  that  has  very  little  to  do  with  school.  The  Martha  Stewart  web  site  seemed  to 
predominantly  be  a marketing  vehicle  for  Martha  Stewart  with  its  prominent  sponsorship  by  the  “Gap”  and  “BMW”. 

Maybe  much  of  the  Internet  as  Resource  model  is  derived  from  a cultural  identity  of  a graduate  student 
where  there  is  always  work  to  be  done,  assignments  to  complete,  reading  to  do,  papers  to  write,  etc..  Perhaps  it  is 
difficult  to  admit  using  the  Internet  for  pleasure  when  it  seems  more  appropriate  for  a graduate  student  to  use  it  as  an 
academic  resource.  Jan  seems  to  support  this  hypothesis: 

1.  no,  only 

and  it’s  only  a function  of  my  circumstances  in  school 
I think  that  I can  use 
easily  get  lost  on  it 

2.  like  if  you  had  the  same  interview  with  me  in  August 
I think  it  would  be  a very  different  interview 

because  I would  have  the  summer  to  just  be  a human  being 
using  the  web  verses  a graduate  student  under  pressure 

In  her  view  being  a “human  being”  is  not  equated  with  being  a graduate  student. 

Patricia  echoes  Jan’s  feelings: 

1.  I mean,  I do 

I have  used  it  for  pleasure 
but  that  doesn’t  happen  for  very  long  but 
but  that  usually  more  in  like  Christmas  break 
or  in  the  summer  when  I don’t  have  school 

As  a librarian  Laura  seems  to  mostly  look  at  the  Internet  as  a resource  for  other  people. 

1.  u mm,  I’d  say 

wool  now  it’s  probably  daily 

just  because  with  my  work  at  the  education  library  on  campus 
a lot  of  the  materials  we  find 
are  on  the  Internet  for  people  so 

Interviewer:  ok,  so  you  use  it  for  your  job  (working  at  the  reference  desk  at  a library)  when  you’re  working  and 

then  as  far  as  personal  use? 

Laura: 

1 . I’d  say  maybe  once  a week. 

Depends  on  what  I’m  really  interested  in  finding 
If  there’s  something  that  I want  to  find  out 
then  I’ll  use  it  more. 
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She  estimates  that  she  uses  it  about  an  hour  a week  for  personal  use.  Laura  also  uses  the  Internet  as  a 
replacement  for  newspapers  and,  like  Ron,  keeps  up  with  current  events  using  the  Web. 

Ron  is  the  only  participant  in  the  study  that  admitted  to,  as  he  put  it,  “truly  surf’,  following  links  without 
any  particular  purpose  in  mind.  I call  this  Schema  the  Internet  as  Information  Frontier.  In  this  schema,  a person 
chooses  to  spend  free  time  following  links  without  any  particular  purpose  in  mind.  Ron  follows  this  strategy  about 
once  a week,  “Yea,  it’s  probably  like  one  day  a week  when  I’ll  spend  at  some  point  just  flippin’  around  just 
following,  following  the  links.’’  This  usually  occurs  on  Saturday.  There  may  exist  a gender  component  in  the 
responses  in  relation  to  how  one  accesses  the  Web,  where  the  act  of  moving  around  is  more  important  to  males  and 
the  final  destination  is  more  important  to  females.  While  one  could  not  draw  any  general  conclusions  from  this,  the 
data  did  seem  to  support  this  hypothesis  for  the  participants  of  this  study  and  is  exemplified  in  this  exchange  with 
Patricia: 


Patricia:  yea,  I like  to  go  just  straight  to  it 
Interviewer:  right,  without  having  to  go  through  interim  steps 
Patricia:  steps  and  yea 
that  usually 

and  1 think  that  probably  turns  me  off 
more  than  anything  else 
Because  I’m  like  “oh,  forget  it” 

Jan  in  describing  how  she  uses  the  Internet  says,  “so  I’m  kinda  an  observer.”  Although  her  first  experience 
with  the  Internet  was  more  free  floating  and  exploratory,  she  had  not  settled  into  the  Internet  as  Information 
Frontier  as  a regular  part  of  her  Internet  experience.  Nevertheless,  she  anticipated  using  the  web  more  freely  in  the 
summer  when  she  didn’t  have  the  responsibility  of  being  a graduate  student.  It  would  be  interesting  to  be  able  to 
watch  the  participants  surf  the  Internet  without  the  expectations  presented  by  an  interview.  Would  female 
participants  experience  the  Internet  as  Information  Frontier  schema  more  than  they  would  admit? 

Space 

An  important  Internet  component  for  all  participants  was  the  Internet  as  Interesting  Visual  Space  schema. 
All  of  the  participants  expressed  in  various  ways  how  they  were  more  attracted  to  as  Laura  put  it,  “visually 
interesting  material”.  Ron,  for  example,  uses  visuals  that  he  is  attracted  to  as  queues  to  look  deeper  into  information 
at  a site.  In  talking  about  the  CNN  site  he  notes: 

1.  At  a glance  you  can  see 

well  the  headline  news 
and  if  anything  is  happening 
that  I'm  interested  in 
1 can  go  right  there  and 
and  to  get  more  details  right  away 
and  eventually  see  a video  clip 
and  some  visuals 
and  news  about  that 


In  general,  all  four  participants  did  not  want  to  use  the  Internet  to  read  lots  of  text.  They  expressed  a 
particular  dislike  for  sites  that  are  text-heavy.  Laura  says: 

1.  because 

if  a site  has  really,  umm 
to  much  text 
or  too  boring 

then  1 probably  won’t  stay  there  very  long 
which  is  not  necessarily 
a reflection  on  the  content 

Jan  answers  a question  about  her  favorite  web  site  with: 

1 . there’s  a media  literacy  site 

that  I absolutely  love 
cause  it  has  the  most  cool  graphics 
and  it’s  so  much  fun 

She  also  dislikes  sites  with  too  much  text: 

1.  I’m  not  at  all  enthralled  by  real  textual  pages 

I kind  of  find  em  to  be  really  dry 
and  dull 

and  I really  just  find  them 
dry  and  dull 
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It  should  be  noted  that  computer  monitors,  depending  on  their  quality,  often  make  it  difficult  to  read  a great 
deal  of  text  and  can  easily  cause  eyestrain.  It  is  difficult  to  say  whether  or  not  this  is  the  cause  of  the  consistent  call 
for  less  text  and  more  graphics,  although  some  of  the  responses  indicated  that  it  has  had  some  bearing  on  this 
attitude.  Laura  also  communicated  a dislike  for  text-heavy  sites  and  addressed  the  difficulty  of  reading  from  a 
computer  monitor  below: 

Interviewer:  Can  you  think  of  a particularly  bad  site  that  you  didn’t  like? 

Laura: 

1 . boy,  lately 

I’ve  had  to  do  a lot  of  readings 
for  my  library  classes 

a lot  of  reserve  readings  have  been  on  the  Internet 
that  have  just  been 
you  know 
full  text  things 

2.  and  umm  and  so  those  are  really 
I mean 

that’s  probably  just  the  exercise  of  reading  off  the  screen 
for  pages  and  pages 
you  know 

Participants,  in  general,  wanted  to  be  able  to  easily  jump  to  the  information  that  they  wanted.  This  schema  is 
called,  Internet  as  organized  Information  Space.  Ron  expressed  his  dislike  for  sites  that  had  search  features  that 
don’t  work  as  he  expects  them  to: 

1 . Sometimes  you  go  into  a company’s  site 
and  there  will  be  a search  button 

implying  that  you  can  just  search  the  site  with  this  one  button 
and  find  a couple  of  links  pointing  in  the  right  direction 

2.  But  you  click  on  a search  button 

and  it  either  says  “not  working  right  now  “ 

or  it  just  doesn’t  point  you  to  where  you  want  to  go  anyway 

so  that’s 

that’s  a real  turn  off  there 

when  features  don’t  work  like  you  expect  them  to 

The  last  stanza  echoes  a common  criticism  that  content  presentation  on  the  Internet  is  still  inconsistent  and 
that  standards  still  do  not  exist  in  terms  of  individual  expectations  and  larger  medium  wide  expectations.  The 
technology  and  techniques  with  which  to  organize  data  on  the  Internet  are  still  quite  early  in  their  development. 

Patricia,  referring  to  why  she  likes  the  Martha  Stewart  site  notes: 

1 . you  go  to  her  home  page 

and  then  I look  at  the  tabs  that  say 
you  know  television,  magazine 
I don’t  remember 
blah  blah,  blah 

and  then  you  can  click  to  that  and  go  to  it 
and  then 

and  then  usually  you  can  make 
like  one  more  click  to  get  to  the  information 
Like  if  it’s  a recipe  on  I don’t  know  some  kind  of  food 
Later  she  says: 

1 . I think  what  turns  me  off 

when  I have  to  keep  going  layer  after  layer  after  layer 
just  to  get  what  I want 

When  asked  about  a site  that  she  particularly  liked  Laura  described  her  use  of  Yahoo,  a reference  site  that 
organizes  listings  of  millions  of  web  sites  by  subject.  She  says: 

1 . I used  to  use  Infoseek  and  Excite 

as  the  web  engines 

and  then  I finally  realized  that  Yahoo 

is  much  better  for  the  type  of  searching  that  I like  to  do 

the  way  it  groups  it 

Later  she  states,  "I’m  interested  in  what  kinds  of  sites  people  find  helpful,  user  friendly."  Laura  helps  library 
patrons  find  information  on  the  web.  When  asked  about  a favorite  site  other  than  a search  engine  or  directory  site 
she  described  a peanut  butter  site  saying: 

1 . it’s  just  that  the  graphics 

the  graphics  are  nice 
but  not  overwhelming 
and  the  text  is  simple 
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2. 


and  1 guess  I like  that  kind 
not  too  much  text 

at  least  at  first  and  then  the  categories  are 
it’s  easy  to  understand, 
it’s  easy  to  navigate 
and  um  it’s  fairly  quick  as  well. 

Included  with  her  perspective  is  the  Internet  as  Interesting  Visual  Space  schema: 
1 . I guess  it  was  jazzy  enough,  too 

jazzy  is  also  a bad  word 

but  umm,  nice  enough  looking  to  keep  my  interest  as  well 


As  exemplified  in  this  answer,  Laura’s  answers  were  often  couched  in  self-doubt.  She  was  not  confident  in 
the  opinions  that  she  presented  about  the  Internet.  If  one  just  looks  at  her  triple  major  as  an  undergraduate 
(Secondary  Education,  German,  and  Political  Science)  and  from  conversations  before,  during  and  after  the  interview, 
it  is  apparent  that  she  is  an  intelligent  person  with  a broad  liberal  arts  background.  It  was  not  immediately  apparent 
what  was  operating  to  cause  this  doubt,  whether  it  was  something  about  my  presence  as  an  interviewer,  or  something 
unique  about  the  Internet  that  challenged  her  confidence  as  an  informed  consumer  and  critic,  or  something  about  her 
own  confidence,  in  general. 

The  Internet  raises  new  issues  about  the  perceived  authority  of  the  content  within  a web  site.  Typical 
measures  of  validity  (i.e.  books  produced  by  established  publishers)  do  not  exist  and  individuals  must  judge  content 
according  to  more  individualistic  criteria  influenced  by  schemas  and  cultural  models.  Who  creates  the  information 
spaces  on  the  Internet  (i.e.  authorship)  seems  to  take  on  a different  meaning  compared  to  other  sources  of 
information  like  books. 

Jan  was  the  only  person  who  directly  spoke  of  the  authorship  of  a site.  Speaking  of  the  media  literacy  site 
that  she  likes,  “somebody  very  sophisticated  put  this  together.”  The  perception  from  participant's  answers  is  that  a 
web  site  is  not  created  by  a person  like  a book,  but  is  created  by  the  organization  or  institution  that  hosts  the  site. 
Unless  the  site  actually  states  the  author’s  name,  it  seems  as  if  people  don’t  conceive  of  authorship  in  the  typical  text- 
based  author/reader  relationship.  The  content  is  just  all  “out  there”  , loosely  connected  but  part  of  a whole  with 
indistinguishable  authors.  Patricia,  when  describing  using  the  web  for  school,  does  pay  attention  to  how  “legitimate” 
information  is  before  using  it  in  a paper.  For  her,  information  that  is  produced  by  governments  is  considered 
“legitimate”.  In  doing  a paper  about  literacy  in  Brazil  she  said,  “and  then  we  ended  up  using,  like,  a lot  of,  like, 
government  sites.” 

Jan’s  fine  arts  and  graphic  arts  background  gave  her  authority  to  describe  the  media  literacy  site  that  she 
liked  in  more  technical  graphic  design  terms.  She  was  quite  confident  in  using  this  social  language.  It  also  was 
apparent  that  she  is  accustomed  to  dealing  with  people  who  don’t  share  the  same  social  language.  She  goes  to  extra 
lengths  to  explain  to  me  what  she  likes  about  the  media  literacy  site  in  stanza  1 below.  In  stanza  2 she  speaks  of  the 
graphics  in  the  media  literacy  site  from  the  perspective  of  an  artist  and  finally  in  stanza  3,  she  speaks  of  her  dislike  of 
animated  icons  on  the  web  and  makes  a claim  that  “most  people  find  them  distracting.”  What  gives  her  the  authority 
to  make  this  claim?  Perhaps  her  background  in  advertising,  according  to  her,  gives  her  insight  into  what  people  like? 

1.  I think  ...wool 

going  back  to  that  web  site  on  media  literacy 
what  drew  me  was  colors 
and  the  energy 

but  I also  think  what’s  really,  REally,  REALLY  important  is  simplicity  and  white  space 
that’s  what  we  call  it  in  graphic  design 
where  this 
there’s  a place 

there’s  a space  for  the  eye  to  flow  around 

rather  than  just 

putting  too  much  on  the  page 

I think  that’s  hard  to  visually  read 

umm  so  I think  you  need  to  have  things  really  clean 

2.  and  that  doesn’t  mean  that  they’re  not 

like  this  this  illustration  on  this  media  literacy  page 

was  very  kinetic  and  energetic 

but  then  around  it  it  was  real  clean 

and  so  it  was  almost  like  a painting  on  a wall 

and  you  had  a sense  of  just 

of  space  of  it’s  own 

so  even  of  itself  it  was  busy 

the  whole  page  wasn’t  busy 
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3. 


a pet  peeve  of  mine 
from  a design  standpoint 
is  those  icons  that  do  things  for  no  purpose 
you  know,  spinning  things  or  flashing  things 
I just  want  to  like  ZAP  them 
I find  them  really  distracting 
1 think  most  people  find  them  distracting 
it’s  really  bad  design 

While  the  participants  were  not  asked  if  they  use  the  Internet  to  communicate  with  other  people,  Jan  is  the  only 
person  who  raised  the  issue: 

1 . I remember  being  able  to  leave  notes  for  people 
because  I was  trying  to  find  out 
at  that  point 

I was  looking  for  information  on  Tiger  Woods 
I was  really  curious  about  what  people  thought  of  the  new  ads 
that  had  come  out 


2.  and  I needed  to  get  a couple  of  copies  of  things  that  1 couldn’t  get  my  hands  on 
and  so  I was  like  finding  these  places  where  you  could  leave  notes  for  people 
and  I was  just 
and  I was  so  intrigued 

and  then  I kept  on  trying  to  find  more  places  that  I could  have  a dialogue  with  people 
or  you  know 

find  people  with  common  interests 
it’s  really  engaging 

This  Internet  as  Communication  Space  schema  would  have  been  interesting  to  explore  in  greater  detail 
with  the  participants.  The  Internet  provides  numerous  opportunities  to  communicate  with  strangers  about  specific 
areas  of  interest. 

Time 

According  to  Barbara  Adam  (1995)  everyone  constructs  their  own  meanings  about  time  based  upon  their 
social,  cultural,  emotional,  physical  and  psychological  circumstance  to  create  what  she  calls  "social  time”.  She  notes 
that  western  style  education  has  worked  to  socialize,  habituate,  and  train  people  into  a “clock  time”  approach  to  time 
which,  in  turn,  has  the  effects  of  displacing  the  variety  of  times  that  comprise  the  multi-dimensionality  of  everyday 
life.  The  time  of  clocks  and  calendars  (e.g.  schedules,  deadlines),  ’’clock  time”  is  only  one  aspect  of  social  time. 
Because  the  Internet  is  always  "open",  those  who  use  it  are  not  confronted  by  the  limitations  of  clock  time  and  in 
turn,  are  liberated  to  construct  their  own  unique  forms  of  "social  time." 

While  each  participant  was  asked  whether  they  ever  lost  their  sense  of  time  when  using  the  Internet,  rarely 
did  they  answer  “yes”.  All  the  participants  were  very  aware  of  the  time  they  spent  using  the  Internet  and  did  not  often 
loose  their  sense  of  time.  Even  Ron,  who  spends  one  day  each  week,  “just  surfing”  plans  this  time  deliberately.  For 
him,  as  for  the  other  participants  it  rarely  happens  by  mistake. 

When  it  does  occur  and  they  spend  more  time  than  anticipated,  it  is  because  they  find  something 
unexpected.  For  example,  this  occurred  when  Ron  was  looking  at  Apple  computer’s  advertising  firm,  Chiatt/Day,  to 
see  some  examples  of  their  work.  He  was  drawn  into  the  site  by  the  amount  of  information  it  offers  about  advertising 
and  how  the  company  researches  people’s  buying  habits  and  tastes.  He  spent  “two  or  three  hours”  looking  at  the 
content  that  the  site  provided. 

1 . It  was  actually  insight  into  the  human  psyche 
more  than  anything  else 

or  was  actually  just  fascinating 

they  were  talking  about 

it  was  about  advertising 

but  it  was  also  talking  about  things 

about  demographics  in  there 

and  what  people  are  doing  in  different  stages  of  there  life 
and  how  it  wasn’t  as  crass  as  how  they  manipulate  you 

2.  but  it  was  really  interesting  to  leam  that  sort  of  thing 
it  had  a very  practical  use 

since  it  was  an  advertising  company 
it  was  practically  oriented 
it  wasn’t  just  esoteric 

This  information  allowed  him  to  go  into  Internet  Time,  when  he  either  was  not  aware  of  the  passage  of 
time  or  didn’t  care.  It  would  be  interesting  to  see  under  what  conditions  people  enter  into  Internet  Time.  Again,  I 
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suspect  that  people  go  into  Internet  Time  more  frequently  than  they  might  like  to  admit,  perhaps  because  the 
Internet  occupies  a unique  position  as  both  sources  of  information  and  entertainment.  And  therefore,  it  is  more 
acceptable  to  use  it  as  a source  of  information  than  as  a place  for  entertainment.  Granted,  people  use  television  both 
as  a source  of  information  and  entertainment,  however  the  line  between  the  two  is  less  clear.  Rarely  will  a graduate 
student  use  the  television  to  research  a paper.  It  would  be  interesting  to  explore  how,  if  it  all,  Internet  Time  may 
conflict  with  the  Resource  schema. 

Ron  describes  Internet  Time  like  a “now”: 

1 . If  there  are  not  many  distractions  around  me 

yes,  time  can  fly  by  pretty  quickly 
If  people  are  in  the  room 
or  if  I'm  trying  to  do  something  else 
like  perhaps  do  the  laundry  or  something  like  that 
that  will  interrupt  that  flow 
Later  he  says  more: 

Interviewer:  Do  you  find  that  the  element  of  surprise  is  something  that  intrigues  you  about  sites  in  general?  When 
you  find  something  you  don’t  expect  to  find  and  that’s  what  might  keep  you  there? 

Ron:  Yeah,  I think  that  it  produces,  you  know, 
some  sort  of  rush  in  my  mind 
you  go  somewhere  and  all  of  sudden  you  just 
you  go  into  that  special  time  mode 
where  time  is  passing  by  and  you’re  not  conscious  of  it 
your  just  following  link  after  link  after  link 
and  could  be  reading  and  reading  and  reading 
whereas  you  hadn’t  intended  to  do  that  when  you  got  there 
That  is  exciting  and  it’s  fun. 


Conclusions 

The  study  constitutes  a beginning  for  using  discourse  analysis  as  a methodology  to  understand  the  cultural 
models  that  influence  how  people  navigate  the  Internet.  While  the  sample  studied  for  this  project  was  small,  the  data 
produced  by  the  interviews  produced  distinct  themes  and  schemas  that  could  be  further  developed  with  a larger 
sample  and  longer  interviews  into  articulations  of  more  general  cultural  models.  The  study  functioned  as  a 
productive  pilot  for  future  research  by  showing  that  social  identities  and  social  languages  contribute  to  how  end- 
users  perceive  and  understand  Internet  content,  as  well  as  , how  these  factors  contribute  to  their  sense  of  time  and 
space  on  the  Internet.  For  example,  the  study  revealed  that  factors  other  than  experience  with  the  Internet  influence 
the  way  that  participants  used  and  analyzed  Internet  content.  Professional  experience  with  information  processing  or 
visual  design  or  business  allowed  participants  to  be  more  confident  consumers  through  social  languages  developed  in 
their  professional  life.  In  such  cases,  cultural  models  as  computer  or  Internet  novices  were  contradicted  by  cultural 
models  developed  through  professional  experience.  In  these  circumstances  they  were  able  to  compartmentalize  the 
two  cultural  models  (novice  and  professional)  which  allowed  both  to  exist  simultaneously. 

Using  discourse  analysis  to  understand  how  people  construct  their  Internet  learning  experience  shows  that 
discourses,  social  languages,  and  cultural  models  are  fluid,  dynamic  and  often  contradictory.  Such  a methodology 
provides  insights  for  educational  media  designers  and  researchers  that  help  them  to  understand  that  even  the  best 
designed  media,  using  traditional  instructional  design  techniques,  may  not  be  effective  because  they  fail  to  address 
how  learning  is  a cultural  process  as  much  as  a cognitive  process.  Using  the  techniques  described  above,  educational 
technology  researchers  can  develop  a language  to  account  for  the  cultural  aspects  that  influence  learning  using 
technology.  With  increasingly  multiethnic  and  multiracial  populations  in  many  parts  of  the  world  it  will  become 
more  difficult  to  ignore  how  cultural  models,  based  on  various  forms  of  difference,  impact  learning. 

In  order  to  more  fully  develop  these  ideas  it  will  require  studies  that  ask  more  in-depth  questions  about  the 
participant’s  educational  and  professional  background,  as  well  as,  their  perceptions  about  how  the  Internet  functions 
as  an  educational,  information,  and  communication  space  for  them.  Further  research  might  include  studying  people 
as  they  use  the  web  using  a protocol  analysis.  While  protocol  analysis  present  problems  in  that  thinking  aloud  may 
alter  a person’s  behavior,  it  might  provide  insight  into  the  ways  a person  approaches  constructing  knowledge  on  the 
Internet  that  an  interview  may  never  reveal.  For  example,  people  may  idealize  or  simplify  ways  that  they  use  the 
Internet  in  an  interview,  where  as  actually  using  the  Internet  they  may  feel  less  self  conscious  and  compelled  to 
“make  sense”  of  their  habits.  Even  videotaping  or  monitoring  the  sites  that  a person  visits  while  using  the  Internet 
would  provide  important  information  about  the  way  people  use  the  Internet.  The  computer  can  easily  record  each 
web  site  that  a person  visits  and  the  amount  of  time  that  they  spend  at  a site.  The  combination  of  the  three  types  of 
data  (interview,  protocol  analysis,  and  direct  monitoring  of  usage)  could  provide  a valuable  source  of  data 
triangulation.  Such  a strategy  would  also  be  helpful  in  showing  the  schema  contradictions,  which  often  exist  when 
looking  closely  at  how  people  process  information  and  construct  knowledge. 
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Finally,  it  should  be  noted  that  discourse  analysis  in  the  context  of  educational  media  research  is  not  being 
suggested  as  a means  to  create  more  efficient  forms  of  instructional  design  that  address  just  the  right  cultural  models 
in  just  the  right  ways.  Instead  by  showing  the  complex  ways  that  cultural  models,  social  identities,  and  social 
languages  contribute  to  Internet-based  learning  experience  it  highlights  that  new  design  paradigms  are  needed  that 
work  within  multiple  discourses  across  gender,  race,  ethnicity,  class,  nation,  etc.  These  new  design  models  would 
strive  to  work  within  ambiguity  and  uncertainty  rather  than  strive  for  efficiency.  Discourse  analysis  provides  a set  of 
tools  with  which  to  begin  this  rethinking  process  by  mapping  and  naming  the  complexities  of  this  shifting  terrain. 
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Abstract 

Can  a class  that  never  actually  meets  within  a classroom  have  a classroom  environment?  This  study 
explores  the  perceptions  of  distance  education  students  of  their  classroom  environment  using  the 
College/University  Classroom  Environment  Inventory  (CUCEI).  It  is  important  to  evaluate  the 
environmental  impact  of  the  classroom  on  the  student  so  the  instructor  can  fully  understand  the  dynamics 
of  the  class  which  might  affect  learning  and  instruction.  This  initial  examination  of  the  virtual  classroom 
environment  found  that  a majority  of  the  students  did  perceive  that  there  existed  a classroom  environment 
and  provided  helpful  instructor  feedback.  Further  iterations  are  necessary  to  develop  a deeper 
understanding  of  the  potential  use  of  the  instrument. 

Introduction 

Distance  Learning  is  by  no  means  a new  phenomenon.  However,  the  new  technologies  provide  a twist  to 
distance  learning  that  is  making  it  grow  and  expand  at  overwhelming  numbers.  The  National  Center  for  Educational 
Statistics  reports  that  in  1995,  a third  of  US  post-secondary  schools  offered  distance  education  courses  with  another 
quarter  of  the  population  planning  to  do  so  in  the  next  three  years.  When  you  then  consider  the  rapid  proliferation  of 
internet-based  courses  as  a distance  learning  option  and  then  consider  that  the  World  Wide  Web  (WWW)  has  only 
been  "popular"  for  the  past  5 years,  it  is  indeed  overwhelming. 

While  the  numbers  alone  are  enough  to  amaze  and  dazzle,  what  is  more  interesting  are  the  instructional 
design  and  pedagogical  issues  that  form  the  basis  of  these  courses  (Ritchie  & Hoffman,  1997).  The  technical  skills  to 
build  a course- web  site  and  all  the  accompanying  technologies  are  merely  psychomotor  skills  ranging  from  the 
simple  to  the  highly  complex,  and  many  courseware  packages  now  remedy  the  need  for  instructors  to  worry  about 
developing  those  skills  (Hansen  & Frick,  1997).  Hill  (1997)  lists  some  of  these  key  issues,  which  include  the 
pedagogical,  technological,  organizational,  institutional,  and  ethical.  What  is  missing  from  these  areas  of  research 
focus  is  the  psychosocial  environment  of  the  classroom.  Research  has  demonstrated  that  the  classroom  environment, 
in  terms  of  psychosocial  factors,  accounts  for  appreciable  amounts  of  variance  in  the  cognitive  and  affective 
outcomes  for  students  (Fraser,  1981). 

However,  can  a classroom  environment  which  is  in  fact  not  physically  tangible,  and  it  exists  on  the  "virtual 
space"  be  assessed?  And,  if  there  really  is  such  a thing  as  a virtual  classroom  environment,  can  it  be  assessed  using 
already  proven  techniques  for  classroom  environment  assessment.  That  query  is  the  focus  of  this  preliminary 
examination  of  the  virtual  classroom  environment. 

Background 

Individuals  do  not  act  outside  the  context  of  their  environment.  The  bond  of  this  physical  and  social 
environment  interaction  with  the  individual  has  troubled  philosophers  since  the  time  of  Aristotle  (Huebner,  1980). 
The  question  therefore  consistently  posed  is  how  can  the  environment  serve  to  enhance  the  development  and  learning 
of  the  student? 

The  study  of  the  classroom  environment  accesses  and  assesses  the  shared  perceptions  of  the  instructors  and 
students  situated  in  that  environment  (Fraser,  1989).  The  advantage  of  this  type  of  assessment  over  outsider 
observational  data  collection  is  that  both  students  and  instructors  have  the  long-term  viewpoint  of  the  class  and  are 
not  basing  their  perceptions  on  small  sample  observations.  The  instructors  and  students  also  place  their  assessment  in 
context  with  other  educational  experiences. 

Moos  (1974)  provides  a scheme  where  there  are  three  classifications  to  describe  human  environments.  The 
first  is  the  relationship  dimension  which  describes  the  nature  and  intensity  of  personal  relationship  and  the  extent  to 
which  there  is  mutual  support  and  assistance.  The  second  dimension  is  called  the  personal  development  dimension 
and  concerns  the  degree  to  which  personal  growth  and  self-enhancement  occur  within  an  environment.  The  third 
dimension  is  systems  maintenance  which  measures  how  orderly  and  responsive  the  environment  is  to  change. 

A number  of  instruments  have  been  developed  to  assess  a variety  of  human  environments,  including  k-12 
classrooms,  psychiatric  hospitals,  prisons,  and  residence  halls.  One  instrument  developed  by  Moos  and  Trickett 
(1974)  is  called  the  Classroom  Environment  Scale  (CES)  and  has  been  used  in  many  of  these  environments  even 
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though  it’s  initial  development  was  for  the  secondary  classroom.  DeYoung  (1977)  piloted  a short-form  of  the  CES  at 
the  university  level. 

Another  instrument  developed  specifically  for  higher  education  is  called  the  College  and  University 
Classroom  Environment  Inventory  (CUCEI).  This  instrument  has  seven  scales  which  cover  the  areas  of 
Personalization,  Involvement,  Student  Cohesiveness,  Satisfaction,  Task  Orientation,  Innovation,  and 
Individualization  (Fraser,  1986).  The  scales  determine  the  fit  between  a student's  perception  of  the  actual  classroom 
environment  and  their  preferred  environment.  The  CUCEI  was  developed  expressly  for  the  purpose  of  assessing 
environments  of  higher  education  classrooms  and  was  developed  for  classes  of  30  students  or  less  and  not  intended 
for  large  lecture  classes  or  lab  settings. 

In  possible  conflict  with  the  assessment  of  a classroom  environment  (the  traditional  physical  space)  is  the 
concept  of  the  virtual  classroom,  such  as  courses  offered  over  the  World  Wide  Web.  Through  Internet  classrooms, 
students  and  faculty  can  communicate  asynchronously  and  synchronously.  Learning  on  the  Internet  can  take  the  form 
of  (a)  electronic  mail  (e-mail)  and  electronic  discussion  groups  (listservs  or  chat  rooms);  (b)  bulletin  boards  or 
newsgroups;  (c)  downloadable  course  materials  or  tutorials;  (d)  interactive  tutorials  on  the  Web;  (e)  real-time, 
interactive  conferencing;  and  (f)  infomatics,  the  use  of  on-line  databases  (Kerka,  1996).  These  methods  provide 
usage,  response  and  impact  considerations  different  from  the  traditional  classroom  setting  (Kuehn,  1994). 

The  study  of  classroom  environment  is  critical  if  teachers  want  to  be  able  to  exercise  control  over  the 
environment  to  the  betterment  of  learning.  However,  with  the  growth  of  distance  learning,  it  is  also  important  to 
determine  what  instructor  and  student  perceptions  are  of  the  virtual  environment.  Verduin  and  Clark  (1991)  state  that 
the  separation  of  teacher  and  learner  does  not  allow  for  a truly  shared  learning  experience,  but  Moore  (1973,  1994) 
concludes  that  distance  learners  must  be  emotionally  independent,  self-motivated,  and  more  autonomous  in  order  to 
compensate  for  the  transactional  distance  in  the  distance  classroom.  Initial  examinations  are  being  made  of  student 
perceptions  of  this  learning  environment  (Powers  & Mitchell,  1997).  This  study  showed  that  students  do  perceive 
that  at  least  two  of  Moos’  dimensions  are  present  (personal  development  and  relationship).  The  question  does  remain 
unanswered  as  to  whether  an  effective  person-environment  fit  is  taking  place  and  the  extent  to  which  the  instructor 
might  be  able  to  control  those  factors.  This  paper  describes  a pilot  study  which  makes  an  initial  examination  of  the 
virtual  classroom  environment  in  terms  of  traditional  measures. 

Methodology 

This  study  details  the  results  of  a pilot  study  which  examined  the  classroom  climate  or  environment  of  a 
virtual  course.  Study  participants  were  graduate  students  enrolled  in  three  different  courses  offered  at  a distance  over 
the  World  Wide  Web  (WWW).  Each  of  the  three  courses  was  offered  by  the  same  instructor  and  was  designed 
following  the  same  design  principles.  One  course  dealt  with  the  history  and  theories  of  Instructional  Technology. 

The  second  course  focused  on  information  technology  and  media  literacy.  The  topic  of  the  third  course  was  on  the 
technologies  of  distance  learning.  Twenty  students  were  enrolled  in  these  three  different  courses. 

The  courses  consisted  of  lecture  notes  placed  by  the  instructor  on  the  WWW,  readings  of  seminal  works  on 
the  topics,  classroom  discussion  completed  through  web  conferencing  and  email  discussion  groups,  synchronous  chat 
sessions  and  presentations  made  by  the  students  and  posted  to  the  WWW.  Course  requirements  included  consistent, 
regular  weekly  participation  in  class  discussion  on  the  web  and  response  to  peers'  presentations  and  ideas. 

As  an  exploratory  study  into  the  evaluation  and  assessment  of  the  virtual  classroom  environment,  data  was 
gathered  from  a variety  of  sources.  First,  a traditional  classroom  environmental  assessment  tool  was  used.  Each 
student  (20)  was  mailed  a copy  of  the  College  and  University  Classroom  Environment  Inventory  (CUCEI)  (Fraser, 
Treagust  & Dennis,  1986).  See  Table  1 for  sample  questions  from  the  CUCEI.  The  CUCEI  is  composed  of  49 
questions  which  rate  seven  different  factors  of  the  college  classroom  environment:  Student  cohesiveness; 
individualization;  innovation;  involvement;  personalization;  satisfaction;  and  task  orientation.  Comprehensive 
validation  research  has  been  completed  on  this  instrument  to  confirm  the  internal  consistency  reliability  and 
discriminant  validity  of  the  instrument  (Fraser  et  al,  1986).  Students  rate  each  question  on  a four-point  scale  o of 
Strongly  Agree  (SA),  Agree  (A),  Disagree  (D),  and  Strongly  Disagree  (SD). 
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Table  I:  Sample  Items  from  CUCEI 


Student  Cohesiveness 

5.  Students  know  exactly  what  has  to  be  done  in  our  class. 

7.  All  students  in  the  class  are  expected  to  do  the  same  work, 
in  the  same  way,  and  in  the  same  time,  (scores  reversed) 

Individualization 

8.  The  instructor  talks  individually  with  students. 

1 4.  Students  are  generally  allowed  to  work  at  their  own  pace. 

Innovation 

15.  The  instructor  goes  out  of  his/her  way  to  help  students. 

19.  The  group  often  gets  sidetracked  instead  of  sticking  to  the 
point,  (score  reversed) 

Involvement 

23.  Students  in  this  class  pay  attention  to  what  others  are 
saying. 

25.  Classes  are  a waste  of  time,  (scores  reversed). 

Personalization 

32.  Classes  are  boring,  (scores  reversed). 

33.  Class  assignments  are  clear  so  that  everyone  knows  what  to 
do. 

Satisfaction 

37.  There  are  opportunities  for  students  to  express  their 

opinions  in  this  class. 

39.  Students  enjoy  going  to  this  class. 

Task  Orientation 

44.  The  instructor  dominates  class  discussion,  (scores  reversed) 
46.  Classes  are  interesting. 

This  instrument  can  be  delivered  in  two  different  formats,  preferred  and  actual.  The  questions  as  worded  in 
Tablet  represent  the  actual  format,  i.e.  how  students  perceive  a classroom  environment  to  actually  be.  In  the 
preferred  form,  the  questions  would  be  the  same,  but  the  word  "would"  is  inserted.  For  example,  "Students  would 
know  exactly  what  has  to  be  done  in  our  class.  For  the  purposes  of  this  study,  only  the  actual  form  was  used  because 
the  research  team  wanted  to  determine  if  this  instrument  could  play  a role  in  the  evaluation  of  virtual  classroom 
environments. 

Twenty  students  were  mailed  a copy  of  the  CUCEI,  along  with  an  explanation  letter  and  a postage-paid, 
addressed  return  envelope.  Thirteen  surveys  (65%)  were  returned  in  sufficient  time  to  be  included  in  this  discussion. 
The  data  from  the  surveys  was  entered  into  SPSS  for  analysis.  Sums  were  generated  for  each  environmental  scale 
and  then  means  and  standard  deviations  were  found  for  each  of  these  sums  (the  CUCEI  is  scored  on  a 1-5  scale  with 
1 being  Strongly  Agree,  5 being  Strongly  Disagree  and  3 given  to  no  answer).  Additionally,  for  investigating  the 
usefulness  of  the  instrument  for  virtual  classrooms,  means  were  generated  for  each  survey  item,  as  well  as  frequency 
data  for  each  item  to  determine  the  degree  to  which  students  considered  the  questions  to  be  answerable  from  the 
perspective  of  a virtual  classroom. 

In  addition  to  the  data  collected  with  the  CUCEI,  qualitative  information  from  the  course  was  also  collected 
in  order  to  assist  with  interpretation  of  CUCEI  results.  The  data  included  papers  and  presentations,  discussion 
questions,  on-line  lectures,  peer  discussion  and  feedback,  and  student  reflective  journals  and  time  logs.  All 
qualitative  data  was  content  analyzed  in  terms  of  the  factors  of  the  CUCEI.  This  process  was  completed  to  examine 
what  factors  the  students  and  the  instructor  perceived  as  important  virtual  classroom  climate  issues. 

Results  and  Discussion 

The  information  received  from  administration  of  the  CUCEI  provided  valuable  information  about  the 
classroom  environment  of  a web-based  class.  The  results  of  the  survey  provided  quantifiable  data  to  support  the 
qualitative  information  gathered  and  the  instructor's  impressions.  First,  the  results  which  lead  to  the  viability  of  the 
use  of  the  CUCEI  in  this  context  will  be  examined,  followed  by  a discussion  on  the  student  assessment  of  their 
virtual  classroom  environment. 

Use  of  CUCEI  for  Virtual  Classes 

Of  the  forty-nine  questions  asked  in  the  CUCEI,  one  or  more  students  did  not  answer  15  of  the  questions.  At 
first  glance  that  seems  to  be  a large  percentage  of  the  total  number  of  questions.  However,  it  is  important  to  see  how 
many  students  didn't  answer  these  questions  in  order  to  make  determinations  on  the  viability  of  the  instrument  (See 
Table  2). 

The  researchers  predetermined  that  one  or  two  blank  responses  was  not  a critical  issue.  There  could  be 
numerous  reasons  why  a student  does  not  answer  a certain  item.  When  three  or  more  students  do  not  answer  that 
same  item  ,than  the  time  must  be  taken  to  examine  the  possible  reasons  behind  the  lack  of  response.  Of  the  original 
15  items  which  received  at  least  one  non-answer,  only  6 remained  when  this  qualifier  is  used.  These  six  items  appear 
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to  fall  into  two  different  categories:  lack  of  a tangible,  physical  classroom  to  assess  and  the  nature  of  an 
asynchronous  web-based  environment. 

When  students  participate  in  a web-based  course,  their  "classroom”  could  be  one  of  dozens  of  places.  For 
example,  a student  might  consider  the  computer  lab  where  s/he  works  to  be  his/her  classroom.  A student  who  works 
mostly  or  entirely  at  home  might  consider  that  particular  space  to  be  where  s/he  attends  class.  Other  students  might 
be  able  to  think  in  the  more  abstract  and  consider  the  web  site  itself  to  be  the  classroom  in  which  they  will  function. 
Finally,  other  students  may  not  be  able  to  conceptualize  a classroom  that  does  not  exist  within  four  walls,  has  desks, 
tables  and  chairs,  chalkboard,  and  needs  a good  cleaning. 

Therefore  it  is  not  surprising  that  those  questions  that  appear  to  concern  a physical  classroom  space  were 
unanswerable  to  students.  In  particular,  those  questions  that  deal  with  the  arrangement  of  the  room  and  the  degree  to 
which  the  instructor  moves  around  the  room.  The  mean  score  for  Item  29  (instructor  moves  around  classroom)was 
2.69,  indicating  that  those  students  who  answered  the  item  tended  to  agree  with  the  question.  The  mean  score  for 
Item  34  (seating  arrangement)  was  2.38,  once  again  indicated  that  those  who  responded  agreed  with  the  statement. 
One  example,  and  a typical  example,  of  how  some  students  might  struggle  with  the  concept  of  a virtual  classroom 
that  doesn’t  have  the  characteristics  of  a traditional  classroom  is  represented  in  this  quote  from  a student  who  is 
trying  to  figure  out  where  all  the  discussion  points  fall  together: 

I noticed  that  [another  student]  sent  the  responses  to  her  questions  [the  listserv].  This  is  not  our  chat  line,  do 
we  use  this  address  only  for  our  assignments?  Thanks  again  for  your  help  I am  working  on  getting  a picture,  but  have 
not  been  successful  up  to  this  date. 

The  student  has  had  no  problems  with  the  assignments;  rather  her  frustration  lies  in  figuring  out  where  in 
the  virtual  classroom  she  needs  to  go  to  tell  the  class  as  a whole  something.  This  type  of  struggle  would  not  be 
typical  in  the  traditional  seminar  classroom. 

Table  2 - Frequency  Data  on  Unanswered  CUCEI  Items 


Item 

Scale 

Frequency 

4.  The  students  look  forward  to  coming  to  class. 

Cohesiveness 

3 

13.  New  and  different  ways  of  teaching  are  seldom  used  in  the  class. 

Individualization 

1 

16.  Student  "clock  watch"  in  this  class. 

Innovation 

2 

19.  The  group  often  gets  sidetracked  instead  of  sticking  to  the  point. 

Innovation 

2 

21.  Students  have  a say  in  how  class  time  is  spent. 

Innovation 

2 

25.  Classes  are  a waste  of  time. 

Involvement 

2 

28.  The  instructor  seldom  moves  around  the  classroom  to  talk  with  students. 

Personalization 

5 

31.  It  takes  a long  time  to  get  to  know  everybody  by  his/her  first  name  in  this 
class. 

Personalization 

1 

32.  Classes  are  boring. 

Personalization 

1 

34.  The  seating  in  this  class  is  arranged  in  the  same  way  each  week. 

Personalization 

7 

39.  Students  enjoy  going  to  this  class. 

Satisfaction 

3 

40.  This  class  seldom  starts  on  time. 

Satisfaction 

3 

41.  The  instructor  often  thinks  of  unusual  class  activities. 

Satisfaction 

3 

45.  Students  in  this  class  are  not  very  interested  in  getting  to  know  other 
students. 

46.  Classes  are  interesting. 

Task  Orientation 

1 

1 

The  remaining  troublesome  items  fall  under  the  category  of  the  asynchronous  nature  of  the  classes  (i.e., 
students’  access  course  materials  at  any  time).  These  items  concern  student  going  to  a class  (which  might  be 
construed  as  physically  going)  and  class  starting  time.  The  mean  for  item  4 (students  look  forward  coming  to  class) 
was  2.23  and  shows  that  the  students  who  did  answer  the  item  agreed  with  the  question,  and  the  same  with  Item  39 
(students  enjoy  going  to  class)  with  a mean  of  2.15.  In  terms  of  the  issue  of  class  starting  time  (Item  40),  even  though 
3 students  didn't  feel  as  though  they  could  answer  the  question,  the  mean  of  1.92  indicates  strong  agreement  with  the 


statement.  The  final  item  under  this  category,  involving  unusual  class  activities  (Item  41),  had  a mean  of  2.69,  and 
again  demonstrates  agreement  with  the  statement.  Under  this  category,  each  of  these  items  were  left  unanswered  by  3 
students,  and  the  same  students  each  time. 

E-mail  and  listservs  are  examples  of  asynchronous  communication.  The  message  can  be  read,  studied,  and  a 
response  sent  at  staggered  times  and  at  the  participant’s  convenience.  E-mail  provides  students  direct  contact  with 
each  other  and  the  instructor  without  the  time  restraints  of  office  hours  and  class  time,  or  space  restraints  of  distance 
(Kerka,  1996;  Partee,  1996).  The  flexibility  of  asynchronous  class  structures  superimposed  on  the  responsibility 
necessary  to  function  in  an  asynchronous  environment  may  cause  problems  for  some  students  and  generate 
uncertainty  as  to  when  and  where  they  should  be  doing  class  work,  as  demonstrated  in  the  following  journal  entry  by 
a student: 


1 am  sorry.  I read  your  message  about  not  posting  until  Tuesday  afternoon,  but  I posted  my  15th  questions 
today.  I am  trying  to  get  ahead  because  I am  going  to  be  gone  next  week.  So  sorry.  Let  me  know  if  I need  to  send 
again.  Thanks  for  your  patience.  When  do  we  need  to  be  available  to  chat  tomorrow? 


Although  none  of  what  the  student  worried  about  was  a serious  problem,  the  ability  to  do  class  work  at 
times  other  than  regular  classroom  hours  was  providing  some  dissonance  for  her. 

Overaii,  fne  CUCEi  items  appeared  to  fair  well  in  the  virtual  classroom.  The  two  items  that  were  most 
difficult  for  students  to  answer  were  not  impossible  for  everyone  to  answer.  Furthermore,  the  difficulty  which  might 
arise  from  conceptualization  of  the  virtual  classroom  environment  might  be  alleviated  as  the  prevalence  of  web- 
based  courses  grows  and  student  experience  and  understanding  of  the  environment  also  grows. 


Assessment  of  the  Virtual  Classroom  Environment 

The  CUCEI  assesses  seven  different  factors  of  the  classroom  environment.  On  each  of  these  factors,  an 
individual’s  total  score  of  7 would  indicate  high  satisfaction  (Strongly  Agree)  with  the  environment,  and  a score  of  35 
would  indicate  high  dissatisfaction  (Strongly  Disagree).  Table  3 provides  the  minimum,  maximum,  mean  and 
standard  deviation  score  for  each  scale. 


Scale 

Minimum  Sum 

Maximum  Sum 

Mean 

Std.  Dev. 

Student  Cohesiveness 

9 

23 

14.23 

3.98 

Individualization 

10 

25 

14.07 

4.09 

Innovation 

9 

23 

14.46 

4.33 

Involvement 

9 

19 

14.54 

2.84 

Personalization 

12 

26 

16.23 

3.66 

Satisfaction 

7 

24 

13.15 

4.45 

Task  Orientation 

9 

22 

14.84 

3.46 

For  the  most  part,  the  scores  from  each  scale  are  similar  to  the  others  scales.  The  mean  scores  for  the  seven 
items  indicate  high  agreement  with  the  statements  in  the  survey  depicting  the  classroom  environment.  The  strongest 
degree  of  agreement  appears  to  be  on  the  Involvement  scale.  Students  felt  that  the  instructor  was  involved  with  the 
students  and  that  the  students  were  able  to  be  involved  and  participate  with  each  other.  This  information  from  the 
instrument  provides  invaluable  feedback  because  it  addresses  one  of  the  biggest  fears  both  students  and  faculty  have 
about  internet-  based  courses,  and  distance  education  courses  in  general.  However,  the  ability  of  students  to  adapt  to 
this  environment  and  to  take  the  initiative  for  making  those  connections  among  participants  is  exemplified  in  this 
message  that  a student  sent  as  a welcome  message  (unsolicited)  to  all  other  students  in  the  class: 

I'm  looking  forward  to  working  with  you  all,  those  who  were  in  class  today  and  those  who  I’ll  meet  in  cyberspace. 

Good  luck. 

For  those  students  who  desire  and  need  the  involvement  with  others,  the  virtual  environment  may  require 
and  force  them  to  be  more  active  about  developing  relationships  as  opposed  to  passively  awaiting  others  to  do  so  for 
them. 

It  is  also  interesting  to  note  that  the  other  scale  that  stands  out  is  the  Personalization  scale.  This  scale  not 
only  had  higher  minimum,  maximum,  and  mean  scores,  but  also  four  of  the  seven  items  on  the  scale  were  not 
answered  by  at  least  one  individual.  Two  of  these  four  items  were  the  items  that  had  large  numbers  of  students  not 
responding  (Items  29  and  34).  This  scale  concerns  how  students  assess  the  environment  in  terms  of  responsiveness  to 
them  as  a person,  and  to  what  degree  they  are  able  to  assert  their  personality.  Again,  the  difficulty  some  students  felt 
about  answering  questions  dealing  with  the  make-up  of  the  classroom  might  have  affected  the  overall  results  of  the 
scale.  Students  in  these  three  web-based  courses  spend  a great  deal  of  time  presenting  their  work  to  all  students  in  the 
class  and  receiving  constructive  feedback.  Students  consistently  comment  about  how  much  they  appreciate  hearing 
from  the  peers  about  their  work  and  the  recognition  it  places  upon  their  efforts,  as  well  as  the  way  this  sharing  forced 
them  to  reflect  back  on  their  own  work: 
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Very  impressive  and  detailed  paper.  Your  insight  on  aspects  on  privacy  and  the  Internet  was  very  good.  You  raised 
many  questions  and  forced  me  to  rethink  many  simple  ideas  such  as  the  privacy  US  mail  has  and  the  lack  of 
privacy  Internet  communication  has.  I look  forward  to  learning  from  your  perspective. 

Overall,  the  CUCEI  provided  valuable  feedback  about  the  classroom  environment.  It  placed  the  comments 
collected  by  students  in  another  framework  that  was  more  quantifiable.  As  an  assessment  tool  of  the  virtual 
environment,  it  generally  appeared  to  provide  good  information. 

Conclusion 

The  CUCEI  appears  to  be  promising  in  its  ability  to  assess  the  virtual  classroom  environment  and  provide 
instructors  with  valuable  information  about  student  perceptions  of  the  environment.  One  thing  this  pilot  study  did  not 
do  was  to  complete  a pre-test  of  the  instrument  that  would  provide  students*  preferred  environment.  As  a follow-up 
study,  it  would  be  interesting  to  assess  students*  preferred  classroom  environment  arid  contrast  that  with  the  actual 
environment.  However,  to  understand  the  virtual  classroom  environment  better  and  because  a virtual  classroom 
environment  can  be  in  many  ways  substantially  different  than  the  traditional  classroom,  it  might  be  important  to 
assess  two  different  preferred  classroom  environments.  The  first  administration  would  examine  the  preferred 
classroom  environment  in  general  terms.  The  next  administration  would  examine  the  students*  preferred  distance 
classroom  environment,  followed  by  assessing  the  actual  perceptions  of  the  virtual  classroom. 

Fraser  (1981)  describes  how  classroom  environmental  research  can  provide  a practical  basis  for  aligning  the 
environment  to  make  a better  person-environment  fit.  For  example,  even  though  an  instructor  may  not  choose  for 
pedagogical  reasons  to  make  all  adjustments  to  bring  profiles  in  line,  the  assessments  allow  the  instructor  to  know 
how  much  weight  is  placed  upon  certain  scales,  i.e.,  greater  emphasis  on  student  cohesiveness  than  individualization. 
Finally,  the  use  of  the  assessment  instrument  also  serves  as  tool  to  encourage  all  those  involved  to  develop  greater 
understanding  of  the  virtual  classroom 
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Abstract 

A theoretical  model  that  permits  graduate  students  to  interact  with  adult  learners  in  an  authentic  situation 
by  addressing  a performance  problem — developing  a community  of  technology-using  teachers  by  using 
asynchronous  and  synchronous  performance  solutions — is  examined  with  implications  presented  for  future 
use. 

A body  of  research  suggests  that  instruction  in  computer  usage  may  decrease  computer  anxiety  and  increase 
certain  attitudes  toward  computers,  thus  opening  a window  of  understanding  about  the  possibilities  this  technology 
may  offer  (Loyd  and  Gressard,  1986;  Madsen  and  Sebastiani,  1987;  Maddux,  Johnson  and  Harlow,  1993;  Savenye, 
Davidson,  and  Orr,  1992).  Designing  effective  computer-based  technology  solutions,  however,  continues  to  concern 
both  teachers  and  students. 

The  potential  of  applying  newer  technologies  may  be  hindered  if  access  to  computers  and  peripheral 
equipment,  time,  and  support  are  limited.  Furthermore,  adult  learners  may  experience  disquietude  if  issues  important 
to  them,  such  as,  connections  to  a learning  community,  peer  support  and  a balance  between  active  and  reflective 
modes  of  learning  are  overlooked  (Brookfield,  1991). 

Project  Goal  and  Objectives 

A model  for  the  development  of  a community  of  technology-using  teachers  was  designed  and  evaluated. 
Asynchronous  and  synchronous  performance  solutions  were  employed  that  permitted  graduate  students  to  interact 
with  adult  learners — the  teachers — as  they  addressed  an  authentic  performance  problem.  The  underlying  concept  was 
that  an  integrated  design  that  connects  two  different  groups  of  adult  learners  with  online  resources  and  easily 
transportable  peripheral  computer  devices  might  have  potential  to  contribute  to  the  graduate  students’  future  design 
of  technology-based  solutions  and  simultaneously  deliver  technical  solutions  to  the  teachers.  Both  synchronous  and 
asynchronous  connectivity  between  the  two  groups  was  offered  primarily  online  through  email,  a web-site,  and 
teleconferencing. 

Emphasis  was  placed  on  developing  effective  performance  solutions  (Mager,  1992)  that  would  encourage 
the  teachers  to  experiment  with,  and  employ,  educational  applications  for  computer  peripherals  in  their  instructional 
science  modules.  The  peripheral  devices  included  video  digitizers  with  zoom  and  electronic  white  board  functions, 
digital  cameras,  handheld  PCs  with  database  applications,  and  laptops  equipped  with  science  probes.  The  graduate 
students  were  expected  to  establish  and  maintain  communication  with  the  teachers;  deliver  performance  solutions  for 
using  the  newer  technologies  that  the  teachers  judged  to  be  effective;  formatively  evaluate  the  teachers’  capabilities 
with,  and  attitudes  toward,  newer  technologies;  and  assess  and  reflect  upon  their  work  with  adult  learners  and  as 
adult  learners  themselves. 

Participants 

Ten  public  school  teachers  at  a small  public  K-6  elementary  school  located  in  the  western  region  of  the 
United  States  volunteered  to  collaborate  with  eleven  students  enrolled  in  a graduate  course  that  emphasized 
theoretical  and  practical  aspects  of  planning  for  professional  development  in  technology.  These  Department  of 
Educational  Technology  students  at  a major  public  university  created  an  instructional  design  intended  to  support  the 
teachers’  needs  for  integrating  technology  into  classroom  science  curriculum.  Two  specialists,  a science  specialist 
within  the  school  system  and  a university  evaluator,  also  joined  with  the  educational  technology  instructor  to 
complete  this  learning  community. 
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Performance  Solutions 


During  the  design  phase,  the  graduate  students  analyzed  the  teachers’  technology  needs.  Those  needs 
centered  on  online  usage,  digital  imaging,  digital  data  display  for  large  group  viewing,  and  technology  integration 
into  traditional  curriculum. 

The  graduate  students  focused  on  ways  to  provide  access  to  newer  technologies  that  appeared  relevant  to 
these  identified  needs  of  the  teachers.  They  planned  time  for  their  mentoring  relationship  with  the  teachers,  and  for 
offering  ongoing  support  through  synchronous  and  asynchronous  contact.  They  also  sought  to  determine  which 
aspects  they  might  best  address  without  face-to-face  (F2F)  training.  They  provided  printed  job  aids  and  a web  site 
with  documentation  on  frequently  asked  questions  about  the  identified  technologies  to  offer  ongoing  guidance 
despite  the  diverse  schedules  of  the  two  groups  of  participants.  Identified  technologies  were  placed  at  the  school  on 
long-term  loan  to  increase  access.  Performance  solutions  and  support  for  basic  applications  of  email  management, 
web-based  searching  for  teaching  and  learning  materials,  and  integration  of  newer  computer  peripherals  into  science 
instruction  were  developed.  After  a synchronous  introductory  approach  of  the  project  and  themselves — a F2F 
meeting  during  one  of  the  teachers’  inservice  days — asynchronous  email  contact  and  a dedicated  web  site  were 
offered  to  the  teachers.  Contact  options  also  included  F2F  meetings  and  telephone  calls  between  mentors  and 
mentees,  videoconferencing  through  hardware  and  software  placed  at  both  the  school  and  university  sites,  and 
portions  of  regularly  scheduled  inservice  days  with  graduate  student  teams. 

The  teachers  guided  the  graduate  students  in  developing  performance  solutions  that  would  be  effective  for 
teaching  aspects  of  technology  usage  related  to  science  standards  to  an  adult  audience.  Both  types  of  adult  learners 
could  engage  in  dual  roles  of  learners  and  students  by  developing  a collaborative  environment  through  the  various 
options  for  ongoing  email  messaging  and  continuous  access  to  a web  site  dedicated  to  the  project.  One  university 
student  served  the  dual  role  of  school  technology  coordinator  and  master’s  student,  providing  a valuable,  visible  link 
between  both  groups. 

Financial  Support 

Financing  from  a Dwight  D.  Eisenhower  Professional  Development  Grant  permitted  the  purchase  of  the 
laptop  computers  and  peripheral  computer  equipment,  planned  as  the  beginning  of  an  equipment  pool  available  for 
long  term  loan  for  professional  development  activities  initiated  in  subsequent  offerings  of  the  university  course. 
Funds  also  made  possible  the  employment  of  a part-time  graduate  assistant  to  serve  as  a liaison  between  the 
university  and  school  populations.  A university  professor  with  expertise  in  evaluation  and  science  education  and  a 
science  specialist  from  the  school  system  were  also  employed.  Both  served  to  connect  the  graduate  students  with  the 
science  curriculum  and  resources  relevant  to  the  school  system.  The  university  professor  also  played  a critical  role 
in  design,  delivery,  evaluation,  and  reporting.  The  science  educator  continued  working  with  the  teachers  after  the 
graduate  students’  semester  ended,  by  guiding  the  teachers  toward  more  advanced  uses  of  the  peripherals  to  improve 
their  students’  science  learning. 

Evaluation 

The  project  underwent  formative  and  summative  evaluation. 

Formative  Evaluation 

The  graduate  students  were  responsible  for  formatively  evaluating  the  project  by  gathering  data  to  initially 

measure: 


1.  The  teachers’  ability  to: 

• use  basic  email  functions  on  the  web  browser. 

• locate  relevant  instructional  materials,  particularly  in  science,  using  a web  browser. 

• use  one  or  more  of  the  newer  computer  peripheral  devices  at  a basic  level  for  instructional  purposes. 

• create  instructional  modules  in  science  that  integrate  one  or  more  of  the  newer  computer  peripheral  devices  at  a basic 
level. 

2 The  teachers’  assessment  of  the  performance  solutions  that  the  graduate  students  developed. 

3 The  teachers’  attitudes  toward  computers. 

The  students  were  required  to  design  their  own  evaluation  approaches  for  items  1 and  2.  For  item  3,  they 
were  provided  with  the  40-item  Computer  Attitude  Survey  (Loyd  and  Gressard,  1984)  to  measure  overall  attitude 
and  domains  of  anxiety,  confidence,  liking,  and  use. 

Summative  Evaluation 

The  evaluator  for  the  project  observed  the  university  classes  and  the  students’  evaluative  data;  made  school 
site  visitations;  reviewed  the  graduate  students’,  university  instructors’  and  teachers’  email  correspondence; 
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consulted  with  the  university  instructor;  and  reviewed  the  teachers’  ideas  for  technology-based  science  lessons.  She 
also  had  access  to  the  graduate  students’  written  summaries  of  their  reflections  about  their  work  with  these  adult 
learners — the  teacher.  Their  reflections  were  framed  within  Brookfield’s  (1991)  eight  insights  on  adult  learning.  The 
teachers’  skills  and  knowledge  about,  and  attitudes  toward,  using  newer  technologies  to  teach  science  in  terms  of 
each  participant  and  as  a group  were  also  evaluated.  The  teachers  provided  some  insights  into  their  subsequent  work 
with  their  pupils  as  a result  of  their  initiating  their  new  knowledge  and  skills. 

Discussion  and  Implications 

The  results  of  creating  an  authentic  performance  problem  for  university  graduate  students  with  a selected 
school  site — the  community  of  learners — are  reflected  in  terms  of  the  initial  technology  issues  of  access,  time,  and 
support. 


Access 

The  teachers  responded  to  access  from  mentor-guided,  hands-on  experiences  and  to  newer  equipment  on 
long-term  loan  at  the  school  by  selectively  experimenting  with  the  newer  technologies.  Strong  response  was  shown  to 
searching  the  web  for  science  resources  and  using  the  digital  camera  to  capture  images.  Some  interest  was  shown  in 
web  browser  use  of  email  beyond  the  school’s  intranet  and  converting  digital  computer  signals  to  analog  ones  with  a 
video  converter  for  display  on  a classroom  television.  No  interest  was  shown  in  using  a handheld  personal  computer. 
Although  only  one  teacher  experimented  with  the  science  probes,  his  involvement  with  it,  led  to  him  to  include  it  in 
his  summer  science  program.  Access  to  a practical  context  resulted  in  the  graduate  students’  design,  delivery,  and 
evaluation  of  performance  solutions  customized  for  a specific  audience. 


Time 

Issues  of  time  for  professional  development  remained  a concern  regardless  of  delivery  mode.  Despite 
external  guidance,  busy  teacher  volunteers  may  not  necessarily  engage  fully  with  newer  technologies  when  there  is 
limited  non-teaching  time  set  aside  for  these  activities.  When  they  spent  time  exploring  applications  of  the  newer 
technologies  with  a science  specialist  and  examining  model  lessons,  they,  and  some  teachers  who  joined  the  project 
at  a later  date,  did  generate  ideas  for  integrating  the  newer  technologies  into  their  science  lessons.  These  ideas  have 
potential  to  become  fully  detailed  lessons  to  be  tested  with  their  pupils. 

The  contributions  of  the  new  teachers  at  the  conclusion  of  the  project  indicate  the  positive  impact  that  a 
credible  and  authoritative  teacher  can  have  for  adult  learners  (Brookfield,  1991).  Although  the  sample  size  for  this 
project  does  not  warrant  strong  generalizations,  future  instructional  designers  may  choose  to  study  the  effect  of 
identifying  specialists  for  their  audience  when  developing  and  delivering  technology-based  performance  solutions. 

Support 

Synchronous  support  from  the  university  students  was  the  preference  of  teachers  who  were  novices  at  using 
technology.  So  strong  was  this  preference,  that  some  teachers  only  began  communicating  with  their  mentors  after 
very  traditional  communication  was  used,  such  as  telephone  calls  and  on-site  visits,  despite  repeated  attempts  to 
initiate  online  contact.  One  mentor  and  mentee  experimented  with  a newly  installed  videoteleconferencing  system 
between  the  two  sites,  but  technical  problems  interfered  with  that  process. 

Although  the  university  students  originally  planned  to  stress  asynchronous  use,  the  active  response  that  the 
synchronous  methods  brought  about  encouraged  them  to  emphasize  those  delivery  modes.  Their  early  plan  to 
incorporate  job  aids  as  guides  to  effectively  using  the  targeted  technologies  was  also  retained.  Creating  these  guides 
required  the  university  students  to  analyze  key  features  of  the  peripheral  devices  and  develop  concise,  clear  print- 
based  guides  to  encourage  independent  use  of  the  technologies  by  the  teachers. 


Future  Directions 

Responses  from  teachers  at  the  school  who  completed  the  Loyd  and  Gressard  Computer  Attitude  Survey 
(1986)  indicate  slight  differences  between  participating  project  teachers  and  nonparticipants.  The  teachers  in  the 
project  were  slightly  more  positive  about  liking  computers;  recognizing  their  usefulness;  and  feeling  less  anxious 
about  them.  The  nonparticipating  teachers  were  somewhat  more  confident.  While  no  statistically  based  conclusions 
are  warranted  because  of  the  low  sample  size,  the  results  provide  a window  that  opens  onto  future  examination  of 
participating  teachers’  prevailing  attitudes  of  positive  change.  Further  study  is  also  warranted  to  determine  if  the 
higher  level  of  confidence  shown  among  the  non-participants  is  a factor  of  greater  expertise  or  the  opposite.  If  the 
latter  situation,  it  may  reflect  lack  of  awareness  of  the  challenges  planned  integration  and  technical  issues  can  pose 
for  newer  technology  users. 

This  project  did  result  in  shaping  a theoretical  model  for  professional  development  for  inservice 
technology-based  performance  solutions  that  engages  graduate  students  directly  in  authentic  school-site  situations.  It 

* • '.  563 

533 


also  guided  the  two  principal  university  professors  involved  in  the  project  to  plan  for  a university-based  learning 
community  between  graduate  students  in  an  educational  technology  and  a science  education  course  for  technology 
enhancement  of  K-12  science  curriculum. 

References 

Brookfield,  S.  (1991).  The  skillful  teacher:  On  technique,  trust,  and  responsiveness  in  the  classroom.  San 
Francisco:  Jossey-Bass. 

Loyd,  B.  , & Gressard,  C.  (1984).  The  effects  of  sex,  age,  and  computer  experience  on  computer  attitudes. 
Association  for  Educational  Data  Systems  Journal,  17(2),  67-77. 

Loyd,  B.,  & Gressard,  C.  (1986).  Gender  and  among  of  computer  experience  of  teachers  in  staff 
development  programs:  Effects  on  computer  attitudes  and  perceptions  of  the  usefulness  of  computers. 

Association  for  Educational  Data  Systems  Journal,  18(4),  302-3 11. 

Madsen,  J.  M.  , & Sebastiani,  L.  A.  (1987).  The  effect  of  computer  literacy  instruction  on  teachers’ 
knowledge  of  and  attitudes  toward  microcomputers.  Journal  of  Computer-Based  Instruction.  14(2).  68-72. 

Maddux,  C.  D.,  Johnson,  D.  L.,  & Willis,  J.  W.  (19967).  Educational  computing:  Learning  with 
tomorrow’s  technologies  (2nd  ed.).  Boston:  Allyn  & Bacon. 

Mager,  R.  F.  (1992).  What  every  manager  should  know  about  training  or  Tve  got  a training 
problem”... and  other  odd  ideas.  Belmont,  CA:  Lake. 

Savenye,  W.  C.,  Davidson,  G.  V.,  & Orr,  K.  B.  (1992).  Effects  of  an  educational  computing  course  on 
preservice  teachers’  attitudes  and  anxiety  toward  computers.  Journal  of  Computing  in  Childhood  Education,  3(2), 
31-41. 


Designing  On-line  Instruction  Using  Multidisciplinary  Approaches 


Sandra  Andrews 
Samuel  A.  DiGangi,  PhD 
David  Winograd 
Angel  Jannasch-Pennell,  PhD 
Chong-Ho  Yu,  Ph.D. 
Arizona  State  University 


Instruction  Support  at  Arizona  State  University  is  a multidisciplinary  group  integrating  academia  with 
Information  Technology.  Unique  features  having  to  do  with  the  status  of  computing  at  the  University  have 
led  to  the  establishment  of  the  Instruction  Support  Lab,  which  has  produced  a series  of  successful 
initiatives  regarding  web  courses.  In  particular ; a close  relationship  with  the  College  of  Education  allows 
the  Instruction  Support  Lab  to  offer  a high  level  of  assistance  in  web  course  design  and  technology 
integration  to  faculty  members.  The  IS  Lab,  having  tested  a variety  of  approaches  to  the  creation  of  web 
courses,  has  chosen  two.  Software  packages  are  made  available,  with  training  provided,  and  these 
packages  may  be  enhanced  if  desired  by  individually  developed  websites . A Multidisciplinary ’ real-world 
team  approach,  modified  to  include  instructional  designers,  is  a second  method.  There  are  pros  and  cons 
to  both  approaches;  lessons  learned  have  been  documented.  Web  course  examples  of  principles  discussed 
are  archived  on  the  corresponding  presentation  website,  at  http://is.asu.edu/aect/multi/. 


It  surprised  many  when  Nicholas  Negroponte,  who  in  the  past  has  spoken  of  the  web  as  creating  “a  totally 
new,  global  social  fabric”(1995),  announced  in  Wired  (1998)  the  end  of  the  digital  revolution.  The  implication  was 
not  that  we  would  now  cease  to  use  computers,  but  that  computers,  and  the  changes  they  bring  with  them,  have 
become  commonplaces  of  our  existence.  One  could  argue  with  Negroponte’ s statement,  yet  in  its  very  publication 
this  remark  signals  the  end  of  an  era:  for  one  hallmark  of  this  revolution  has  been  that  a thinker  in  tune  with  the  times 
could  create  new  realities  with  a word  (Davis,  1998;  Rheingold,  1993a).  Logos  has  had  incredible  power  in  these  last 
years:  writers,  thinkers,  educators  could  formulate  a goal  and  then,  thanks  to  electronic  communication,  quickly  find 
a group  of  like  minds  with  which  to  plan  and  implement  means  of  reaching  that  goal.  With  this  in  mind,  it  is  an 
appropriate  time  to  detail  the  history  of  Instruction  Support  at  Arizona  State  University,  sharing  methods  that  might 
be  useful  to  others,  and  sharing  lessons  learned  in  the  first  two  and  one-half  years  of  the  Instruction  Support  Lab. 

The  Instruction  Support  Lab  was  founded  in  the  summer  of  1996,  as  an  open  door  for  faculty  to  the 
Instruction  Support  group  in  central  Information  Technology  at  ASU.  Instruction  Support  itself  came  into  being  as 
part  of  a deliberate  restructuring  of  central  computing  that  brought  an  academic  influence  into  Information 
Technology,  in  particular  from  the  College  of  Education.  This  restructuring  began  with  the  creation  of  an  academic 
position,  the  Vice  Provost  for  Information  Technology,  filled  by  a professor  from  the  Department  of  Computer 
Science  and  Engineering,  and  continued  with  the  creation  of  the  Instruction  and  Research  department.  A professor 
from  the  College  of  Education  was  brought  in  to  head  Instruction  Support,  and  he  was  joined  by  two  more 
researchers  from  the  College  of  Education. 

Much  of  the  Lab’s  development  has  centered  around  the  evolution  of  web  courses  and  web  supported 
courses.  Very  quickly,  users  proliferated.  In  the  summer  of  1997,  before  the  Lab  was  even  one  year  old,  the  Lab  staff 
began  searching  for  ways  to  streamline  development  work.  Caught  between  the  need  to  develop  interactive  websites 
and  the  need  to  serve  faculty  members  new  to  the  web,  the  Lab  staff  began  designing  templates  and  interfaces  so  that 
faculty  members  could  set  up  web  pages,  quizzes  and  bulletin  boards  without  knowing  HTML.  As  other  solutions 
arrived  on  the  market  they  were  tested  and  offered  to  professors  who  were  willing  to  try  them  out.  As  there  was  no 
one  solution  that  provided  all  the  features  we  identified  as  important  in  a web  course,  IS  continued  to  develop  our 
own  applications.  We  also  kept  in  constant  touch  with  what  our  colleagues  in  the  commercial  world  were  thinking 
and  doing  by  following  discussions  on  professional  mailing  lists.  We  had  seen  from  the  beginning  that  real-world 
development  is  always  the  product  of  a multimedia  team  (Nielsen,  1995),  and  we  began  refining  this  concept  for 
academic  purposes. 

Major  Arguments 

That  the  lab  has  not  remained  static,  but  has  been  able  to  adjust  support  for  technology  integration  in  the 
face  of  a technology  that  changes  daily,  is  the  real  hallmark  of  success.  Three  areas  in  particular  in  which  we  can 
point  to  specific  successes  are  first,  new  areas  of  support  in  Information  Technology  that  have  developed  from  lab 
initiatives;  second,  students  who  have  “graduated”  from  the  lab;  and  third,  courses  and  other  projects  developed  in 
Instruction  Support.  Web  courses  and  web-supported  courses  have  been  a central  area  in  terms  of  faculty  requests  for 
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support,  with  over  forty  courses  affected  by  the  Lab.  A number  of  Lab  projects  having  to  do  with  improving  web 
course  development  have  spun  off  into  other  areas.  Each  of  these  projects,  having  grown  too  large  for  the  Lab,  is 
now  being  developed/offered  to  faculty  elsewhere  in  Instruction  Support  and  Information  Technology.  For  instance, 
support  for  those  using  web  course  creation  software  packages  such  as  WebCT  has  moved  out  of  the  lab  to 
Instruction  Support’s  Education  area.  Support  for  collaboration  software,  pioneered  with  faculty  members  in  the  Lab, 
has  also  moved  out  to  a production  environment.  Our  initial  efforts  in  creating  scripts  for  interactivity  resulted  in  the 
funding  of  an  IS  graduate  student  to  complete  a set  of  interfaces  by  which  online  quizzes  could  be  created  and 
administered.  Our  efforts  to  develop  a web  course  creation  package  resulted  in  the  project  being  funded.  Student 
successes  include  three  students  who  are  now  full-time  staff  members  at  ASU,  a student  who  became  one  of  the 
three-member  webmaster  team  at  ASU,  and  students  who  received  excellent  job  offers  upon  graduation  or  earlier. 

Key  features  contributing  to  the  Instruction  Support  Lab 

Four  key  features  that  have  led  to  these  successful  web  courses,  initiatives,  and  students  are,  first,  the 
previously-mentioned  integration  at  ASU  of  academia  and  computing;  second,  a university-wide  policy  of 
decentralization  with  regard  to  computing  resources;  third,  a lab  mission  that  includes  many  levels  of  support  as  well 
as  research  and  development;  and  fourth,  an  openness  in  the  lab  itself  to  real-world  methods.  The  previously- 
mentioned  change  in  the  nature  of  Information  Technology  is  the  first  key  feature  affecting  the  Lab.  At  its  inception 
the  IS  Lab  replaced  a modest  lab  where  faculty  came  for  tasks  such  as  printing.  The  new  academic  focus  made  it 
possible  for  representatives  of  Information  Technology  who  understood  the  needs  of  faculty  to  reach  out  to  the 
Colleges;  improved  access  for  faculty  members  to  IT;  and  allowed  IS  Lab  staff  members  to  advise  faculty  on  good 
instructional  design.  In  the  past,  no  web  course  support  had  been  offered  other  than  to  one  law  professor  who  had 
come  to  the  faculty  lab  to  learn  HTML.  Now  that  same  professor  was  offered  instructional  design  support;  help  with 
graphics  that  were  carefully  crafted  to  enhance  instruction  while  simultaneously  following  the  rules  of  good  web 
design;  and  interactivity.  An  invitation  to  collaborate  in  research  could  also  be  extended  since  Instruction  Support 
now  had  researchers  to  evaluate  major  projects  and  present  the  results  at  academic  conferences,  or  publish  them  in 
scholarly  journals. 

The  second  key  feature  affecting  the  Lab's  success  is  that,  as  ASU  Provost  Milton  Glick  (1999)  has  stated, 
Arizona  State  University  is  a decentralized  campus  with  regard  to  computing  resources.  We  in  Instruction  Support 
feel  it  is  inappropriate  to  mandate  technology  solutions  across  the  board  where  instruction  is  concerned,  just  as  it 
would  be  inappropriate  to  mandate  that  all  professors  use  the  same  instructional  strategies  regardless  of  academic 
discipline.  In  1999,  solutions  for  instructional  problems  are  still  being  developed;  no  one  solution  can  be  relied  on  to 
take  care  of  all  needs,  whether  it  is  a web  course  management  system,  an  instructional  strategy,  or  a particular 
platform.  In  keeping  with  this  position,  ASU  has  allowed  differing  support  structures  to  arise.  Currently  there  are 
three  primary  support  structures  for  web  courses.  ASU  professors  desiring  support  can  work  with  Information 
Technology's  Instruction  Support,  with  the  College  of  Extended  Education,  or  with  a new  entity,  the  Center  for 
Learning  and  Teaching  Excellence.  Each  has  a different  focus;  faculty  members  can  use  more  than  one  if  they  desire. 
Instruction  Support  offers  an  individualized  approach  that  allows/requires  the  faculty  member  to  participate  in 
development.  Extended  Education  is  our  distance  education  arm  at  ASU;  faculty  using  their  services  may  supply  a 
course  already  developed  by  the  professor,  supply  oniy  the  content  and  have  the  graphics  developed,  or  experiment 
with  “canned”  content.  The  Center  for  Learning  and  Teaching  Excellence  is  expected  to  serve  as  a bridge  between 
the  two,  focusing  on  faculty  development  but  familiar  with  the  changing  offerings  of  other  support  areas. 

The  stated  mission  of  the  Lab  is  the  third  key  feature  influencing  the  success  of  the  Lab.  Its  tenets  derive 
from  the  atmosphere  of  academic  freedom  that  arises  where  a campus  is  decentralized  with  regard  to  technology,  and 
where  Information  Technology  has  an  academic  flavor.  The  primary  mission  of  the  IS  Lab  is  to  help  integrate 
technology  into  instruction.  A faculty  member  might  still  ask  for  help  with  printing,  but  others  have  asked  for  help  in 
planning  the  simultaneous  premiere  of  a new  work  of  classical  music  on  the  web  from  nine  locations;  in  finding  the 
way  for  a department  to  rearrange  itself  into  a team  in  order  to  publish  an  online  as  well  as  a print  version  of  a 
scholarly  journal;  and  in  rewriting  a piece  of  software  so  that  a professor  of  communication  can  create  a web  course 
without  with  maximum  opportunities  for  communication.  Beyond  this,  the  Lab  constantly  conducts  its  own 
investigations  into  new  technologies,  so  as  to  be  prepared  with  advice  when  faculty  approach  us  with  instructional 
problems  as  well  as  to  help  drive  the  evolution  of  these  technologies  in  the  directions  we  as  academics  need  them  to 
evolve. 

Finally,  a fourth  key  feature  contributing  to  success  is  the  use  of  real-world  development  methods,  always 
tempered  and  enriched  by  the  academic  viewpoint.  Like  labs  on  many  other  campuses,  we  have  experienced  the 
heady  feeling  of  having  the  best  of  two  worlds,  academia  and  the  commercial  “real”  world.  In  the  academic  tradition 
we  have  the  freedom  to  conduct  research  and  the  luxury  of  supporting  worthwhile  projects.  In  common  with  the 
commercial  world  we  have  the  realization  that  we  are  fortunate  to  live  in  a time  in  which  our  shared  intelligence  can 
find  solutions  and  new  directions  almost  as  often  as  we  need  them.  Developers  who  are  in  the  forefront  of  web 
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design,  digital  artists,  and  many  of  the  best  instructional  design  minds  all  share  their  insights  with  us  through  online 
conversations  and  collaboration. 


Real-world  methods 


The  most  important  real-world  method  in  use  in  the  IS  Lab  is  the  concept  of  the  multidisciplinary 
/multimedia  web  development  team.  No  single  person  can  create  any  kind  of  multimedia  or  web  instruction  alone  if 
that  instruction  is  to  compete  visually  and  professionally  with  commercial  products.  It  would  be  impossible  for  the 
individual  faculty  member  to  simultaneously  be  a professional  in  all  of  the  areas  needed  in  development.  Thus  the 
faculty  member  coming  into  the  lab  to  develop  a web  supported  course  becomes  one  member  of  a development  team, 
in  this  case  the  content  expert  as  well  as  a project  co-manager.  A second  project  manager,  equal  in  authority, 
represents  the  Lab.  As  needed,  a graphics  expert,  a videographer,  a programmer,  and  an  instructional  designer  may 
be  assigned  to  the  project.  Each  team  member  is  a student,  and  all  bring  the  latest  information  from  their  respective 
fields.  The  lab  is  a crucible  in  which  these  students  from  different  disciplines  must  learn  to  work  together.  The 
graphics  designer  learns  quickly  that  small  file  sizes  are  important  to  the  web  designer;  the  programmer  learns  to 
appreciate  that  it  takes  as  much  dedication  and  as  many  late-night  hours  to  create  visual  perfection  as  it  does  to 
create  the  perfect  program.  As  student  team  members  learn  to  respect  one  another  they  realize  the  invaluable 
experience  they  are  gaining:  they  will  be  prepared  for  the  development  teams  that  exist  in  their  future  places  of 
employment  (Crandall  & Levich,  1998;  Ullman,  1997.)  The  supervisor  becomes  the  bridge  among  students  from  the 
various  disciplines.  In  all,  the  Instruction  Support  Lab  has  a staff  of  twelve  student  workers,  from  as  many  of  the 
University’s  Colleges  as  possible.  While  the  majority  are  students  in  Education,  Engineering,  Fine  Arts,  and 
Architecture,  students  from  all  disciplines  have  something  to  offer. 

A second  real-world  method  is  the  concept  of  the  “flat”  hierarchy.  This  concept  is  prevalent  in  what  Pearce 
(1997)  calls  “virtual”  companies:  small,  low-overhead  companies  that  can  expand  and  contract  as  needed.  Downes 
and  Mui  (1998)  report  the  concept  as  a successful  strategy  in  development,  quoting  a CEO  as  asking,  “How  do  you 
change  a culture  from  one  of  hierarchy  with  the  normal  pyramid  to  an  open,  flat  culture  where  the  25-year-old  can 
say  to  me,  ‘You’re  crazy.  That  won’t  work.  Where  did  you  get  that  dumb  idea?”  A flat  hierarchy  does  not  mean  that 
the  CEO  is  no  longer  the  CEO,  but  that  a system  of  revolving  authority  can  allow  the  expert  in  a field  to  make  the 
decisions  involving  that  field.  The  expert  must  be  willing  to  share  information,  and  the  other  team  members  must  be 
willing  to  learn,  so  that  responsibility  for  decision-making  stays  with  the  expert  only  so  long  as  the  decisions  involve 
the  expert’s  field.  In  practice,  this  implies  that  the  supervisor  is  constantly  being  taught  by  the  employees,  so  that  the 
supervisor  can  participate  in  decision-making  regarding  the  employees  from  an  informed  standpoint;  and  it  gives 
employees  a chance  to  be  committed  and  to  do  their  best. 

A third  real-world  method  is  that  of  allowing  staff  members  to  manage  their  time.  Student  workers  in  the  IS 
Lab  are  told  upon  being  hired  that  they  will  be  expected  to  spend  approximately  half  their  time  on  lab  tasks  shared 
with  other  Lab  staff:  working  with  drop-in  clients,  scanning,  documenting,  even  dusting.  The  other  half  of  their  time 
should  be  spent  in  activities  which  will  help  them  develop  skills  in  their  area  of  studies  and  interests.  There  may  be 
times  when  as  much  time  as  possible  will  be  devoted  to  a project  deadline  that  demands  that  everyone  contribute. 
However,  this  will  be  followed  by  time  for  the  student  to  learn  a new  software  program  or  to  work  on  a project  of 
interest. 


Fourth,  the  Lab  uses  a system  of  networking  and  internships  to  find  qualified/competent  student  staff. 
Because  our  work  environment  is  unique  on  the  ASU  campus,  the  Lab  has  no  trouble  attracting  applicants,  yet  we  do 
most  of  our  hiring  through  networking  with  former  employees  often  recommending  new  students.  These  new 
students  usually  apply  first  as  unpaid  interns,  willing  give  their  time  for  a chance  to  learn  the  skills  of  working  in  a 
multidisciplinary  team  and  creating  professional  work.  Such  interns  are  given  the  responsibilities  of  full  Lab  staff 
members.  Interns  are  required  to  keep  a journal  and  to  produce,  or  work  as  a team  member  on,  one  large  project  that 
becomes  a part  of  their  portfolio  and  is  useful  to  the  lab.  In  addition  to  interns,  the  Lab  has  a mentoree  program,  in 
which  young  adults  who  would  not  otherwise  have  access  to  technology  are  given  the  status  of  interns  in  the  lab.  Of 
our  four  mentorees  thus  far,  one  was  a learning-disabled  young  man;  two  came  to  us  through  a technology  mentoring 
program  for  the  formerly  homeless;  and  one  is  deaf. 

Fifth,  the  Lab  relies  on  collaborative  procedures  involving  electronic  and  face-to-face  meetings,  for  the 
iterative  nature  of  the  developmental  process  requires  collaboration  (Yourdan,  1998).  All  students  are  given  accounts 
on  the  Lab  servers;  after  a recent  survey  of  internal  files  on  our  main  server  it  was  estimated  that  almost  one  quarter 
were  uploaded  for  collaborative  purposes.  Students  working  on  a project  may  not  be  in  the  Lab  at  overlapping  times, 
so  collaborate  by  uploading  their  work  to  a shared  directory  on  the  server,  then  notifying  the  other  students  via  e- 
mail  of  the  project  status.  A particularly  important  collaborative  procedure  is  the  “studio  model”  adopted  from  the 
College  of  Architecture.  Under  this  model,  students  develop  in  a lab  environment,  and  a team  member  is  encouraged 
to  request  feedback  from  peers  at  any  time  in  the  development  process,  even  if  they  are  not  on  the  same  team.  An 
extension  of  this  is  the  concept  of  the  small  usability  study,  in  which  an  interface  is  tested  by  a group  of  seven  to 
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thirty  users.  In  addition,  we  are  currently  using  collaboration  software  in  order  to  keep  documentation  and  student 
workload  information  easily  accessible. 

Finally,  as  far  as  possible,  the  Lab  uses  real-world  planning  and  documentation  procedures.  These  can  be 
difficult  to  implement,  as  neither  professors  nor  students  may  see  the  use.  However,  the  use  of  a multidisciplinary 
team  that  does  not  always  meet  face  to  face  requires  the  use  also  of  planning  documents  such  as  timelines  and 
storyboards.  Such  methods  contribute  to  the  success  of  commercial  companies,  as  they  break  projects  down  into 
smaller  tasks  and  issues  (Kristof,  199);  thus  they  can  only  improve  our  instruction.  While  developers  agree  that  a 
project  management  tool  that  meets  all  needs  has  not  yet  appeared  (England  & Finney,  1999),  as  better  software  for 
project  management  is  developed,  documentation  will  become  an  expected  part  of  project  development  for  all  team 
members. 

Streamlining  procedures 

Streamlining  goes  hand  in  hand  with  technological  sophistication  (Porter,  1997)  and  is  necessary  if  the  level 
of  sophistication  is  to  be  maintained.  One  means  of  allowing  the  Lab  to  remain  dynamic  in  the  face  of  changing 
technologies,  numbers  of  users,  and  levels  of  user  ability,  has  been  the  Lab’s  relationship  to  other  areas  within 
Instruction  Support.  Large  projects  may  require  support  from  staff  members  in  production  areas.  Similarly,  Lab  staff 
members  initiate,  evaluate,  and  test  new  ideas  in  a development  environment;  successful  initiatives  and  ideas  are  then 
moved  to  a production  environment  as  projects  grow.  This  streamlining  procedure  ensures  that  Lab  staff  are  then  free 
to  begin  new  initiatives.  With  web  course  development  we  focus  on  finding  ways  to  streamline  procedures  while 
maintaining  high  standards  in  terms  of  interface  design,  programming  issues,  web  tenets,  instructional  design,  and 
learning  theory,  while  allowing  maximum  individualization  for/  input  from  professors.  This  balance  ensures  that  we 
keep  current  with  information  and  with  demand.  A far  as  possible,  web  course  development  falls  within  a set  of 
overarching  ideas,  philosophical  and  instructional.  This  set  of  standards  frees  staff  developers  to  experiment  and 
bring  in  new  ideas  so  long  as  the  ideas  remain  within  this  set  of  guiding  parameters.  For  hyperlinked  examples,  see 
the  website  at  http://is.asu.edu/aect/multi/  . The  guidelines  are  as  follows: 

1 . The  professor  is  an  integral  part  of  the  design  team,  not  just  the  subject  matter  expert. 

2.  Development  should  provide  for  maximum  reusability  of  procedures,  templates  and  tools  developed  for  web  courses.  Any 
such  products  developed  for  a course  should  be  designed  so  as  to  be  easily  configured  for  other  courses. 

3.  We  attempt  to  apply  principles  from  communication,  learning  theory,  commercial  multimedia/video  production,  and 
distance  learning. 

4.  We  value  content  first,  using  technology  only  where  it  is  appropriate. 

5.  We  seek  to  design  for  all  users.  We  value  the  accessibility  standards  developed  by  the  World  Wide  Web  Consortium 
(1998).  The  electronic  medium  can  provide  disabled  students  with  a chance  to  use  their  abilities  and  creativity  (Barnette, 
1994)  if  developers  adhere  to  those  standards.  Whenever  possible,  we  use  multi-platform  solutions;  we  keep  file  sizes  and 
screen  sizes  to  a level  that  as  many  users  as  possible  can  view;  we  use  few  passwords,  preferring  to  share  information 
when  possible;  and  we  attempt  to  use  good  HTML  so  as  to  be  viewable  everywhere. 

6.  Those  using  the  lab  resources  are  asked  to  give  back  to  the  Lab  in  some  way.  Often  a professor  places  a graduate  assistant 
or  student  worker  in  the  lab. 

7.  Whenever  possible,  standard  multimedia  documentation  procedures  are  used.  We  encourage  professors  to  make  plans  with 
us  for  both  formative  and  summative  evaluation. 

8.  We  pay  particular  attention  to  visual  communication  in  interface  design.  This  is  an  aspect  of  web  course  development  that 
many  course  designers  have  been  slow  to  take  into  account.  This  may  be  because  interface  design  is  not  seen  as  being 
particularly  important  by  professors  who  are  used  to  considering  only  content  when  designing  a face-to-face  course.  Many 
web  courses  are  consequently  still  content-only,  or  use  standardized  icons  for  navigation  that  are  the  same  across  courses. 

It  is  our  belief  that  visual  communication  interface  design  does  indeed  have  a counterpart  in  the  face-to-face  world.  The 
classroom  itself  is,  after  all,  an  interface  (Jonassen,  Davidson,  Collins,  Campbell  & Haag,  1995).  Students  learn  in  an 
environment  with  which  they  interface  using  all  senses.  One  course  may  take  place  in  a large  mediated  classroom  with 
projection  screens;  another  may  take  place  in  a small  room  with  rows  of  desks.  Ignoring  these  aspects  of  the  environment 
produces  difficulties  for  those  attempting  to  learn  or  teach  via  the  web  (Fleming,  1998;  Mandel,  1997).  Sensory  cues  are 
bound  up  with  their  learning  and  help  students  remember  what  they  have  learned. 

9.  We  also  pay  particular  attention  to  navigation  issues.  Confusing  interfaces  slow  down  the  student,  who  will  have  to  adjust 
to  the  interface  before  learning  can  begin  (Fowler,  1995;  Galitz,  1997).  Commercial  websites  and  television  may  be 
perceived  of  as  noise-free  by  viewers,  since  research  is  used  to  produce  the  optimum  visual  environment.  Students  have 
become  accustomed  to  receiving  information  in  a particular  format,  and  information  given  in  another,  less  professional 
mode  may  be  difficult  to  remember.  This  is  not  to  say  that  educators  need  to  become  entertainers.  Educational  television 
uses  the  design  principles  of  commercial  television,  and  thus  successfully  transmits  content.  Web  courses  need  similarly  to 
investigate  how  information  is  transmitted  in  the  commercial  world,  in  order  to  avoid  distracting  users. 

Methods  of  producing  webcourses  in  Instruction  Support 

Professors  and  instructors  requesting  support  with  the  creation  of  web  courses  are  offered  a choice  of  two 
major  methods  for  creating  those  courses:  first,  they  can  choose  to  use  a web  course  creation  software  tool,  and 
second,  they  can  choose  to  become  a part  of  an  individualized  multimedia  team.  The  two  methods  can  also  be 
combined  as  desired.  The  professor  who  wishes  to  create  a web  course  is  asked  to  attend  an  initial  meeting  with 
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representatives  of  the  Lab  staff.  Here,  levels  of  possible  development  are  discussed.  The  professor  may  choose  (a)  to 
create  a basic  level  web  supported  course,  with  syllabus  and  class  schedule;  (b)  to  create  an  interactive  course  using 
a software  tool;  or  (c)  to  create  an  interactive  course  that  will  be  designed  from  the  ground  up.  If  the  faculty  member 
wishes  to  use  a software  solution,  the  course  is  moved  to  the  IS  Education  area,  where  training  is  provided.  If  the 
faculty  member  wishes  instead  to  develop  with  a multidisciplinary  team  of  Lab  staff  members,  then  the  faculty 
member  becomes  project  co-manager  and  usually  provides  one  or  more  graduate  students  for  the  team.  Faculty 
members  may  elect  to  work  in  both  ways;  a professor  might  decide  to  develop  a website  in  the  Lab  that  ties  together 
and  individualizes  two  courses  developed,  with  a software  tool,  in  the  Education  area.  Should  the  faculty  member 
decide  to  develop  an  interactive  course  from  the  ground  up,  then  the  multidisciplinary  team  may  include  not  only  IS 
Lab  students,  but  also  IS  staff  members  such  as  programmers  and  videographers  from  production  areas. 

The  multidisciplinary  team  approach  to  course  development 

If  development  is  team-based,  the  professor  is  asked  to  attend  at  least  four  meetings,  so  as  to  be  able  to 
provide  direction;  this  also  ensures  that  communication  is  efficient.  After  this,  a graduate  student  provided  by  the 
faculty  member  may  attend  as  co-manager  in  place  of  the  faculty  member.  Documentation  paperwork,  beginning 
with  the  first  meeting,  guides  the  team  through  a standard  set  of  planning  documents  including  proposal,  content 
outline,  flow  chart,  and  detailed  storyboards.  An  instructional  designer  from  fne  Coiiege  of  Education  is  responsible 
for  helping  team  members  create  these  necessary  documents;  the  documents  will  save  valuable  production  time  later. 
IS  Lab  graphics  experts  work  with  the  instructor  to  create  a series  of  graphics  that  communicate  visually  what  the 
course  is  designed  to  do.  Typically,  a student  from  Fine  Arts  or  Architecture  creates  a series  of  three  sketches,  and 
the  professor  then  chooses  one  line  of  development.  An  programmer,  usually  from  Computing  Science  and 
Engineering,  works  with  the  artist  to  create  the  website.  Interactive  elements  are  brought  in  as  needed.  All  graphics 
must  work  in  harmony;  motifs  are  chosen  to  reflect  the  personality  of  the  instructor  as  well  as  the  nature  of  the 
course,  for  the  course  is  in  some  sense  a creative  work,  and  the  professor  creating  the  course  stands  in  the 
relationship  of  artist  or  writer  to  those  who  access  the  course  (Maddox,  1996). 

We  view  these  initial  meetings  as  an  opportunity  to  learn  the  answers  to  questions  that  we  ask  in  order  to 
improve  our  work.  The  early  meetings  also  provide  an  opportunity  to  help  the  professor  understand  the  nature  of  the 
procedures  used.  We  take  the  time  to  explain  professional  development  methods  (such  as  timelines  and  storyboards) 
to  the  professor.  We  ask  where  professors  heard  about  the  Lab;  this  enables  us  to  plan  for  future  information 
dissemination  about  Instruction  Support.  We  discuss  evaluation  procedures,  suggest  collaboration  in  research,  and 
explain  backup  procedures.  We  also  ask  the  professor  for  the  primary  reason  for  creating  a web  course  or  web 
supported  course.  The  answer  to  this  question  helps  us  direct  development  and  enables  us  to  predict  what  sorts  of 
courses  we  need  to  prepare  to  develop.  For  example,  a number  of  faculty  members  have  come  to  us  because  they 
teach  very  large  classes,  and  feel  that  they  need  new  means  to  make  information  accessible;  for  this  group,  we  may 
suggest  posting  presentation  slides,  lecture  notes,  and  syllabi.  A second  group  of  faculty  members  may  wish  to  create 
a course  for  faculty  development  or  funding  reasons.  A third  group  is  interested  in  the  concept  of  web  instruction  and 
wishes  to  try  out  new  ideas;  for  this  group,  we  can  investigate  the  pros  and  cons  of  those  ideas  and  ways  of 
implementing  them.  Other  faculty  members  have  been  particularly  interested  in  experimenting  with  interactivity,  so 
that  small  groups  can  collaborate  or  so  that  members  of  a small  course  can  work  closely  together. 

The  software  method  and  the  multidisciplinary  team  approach  each  have  strengths  and  weaknesses  that  must 
be  taken  into  account  when  choosing  a method  for  a particular  course.  The  advantages  of  a multidisciplinary  team 
are  that  this  method  (a)  provides  for  maximum  individualization,  (b)  provides  for  maximum  configurability,  (c)  can 
lead  to  a course  that  is  professionally  developed  and  of  the  highest  quality  in  every  area,  (d)  consists  of  well- 
integrated  modules  connected  via  good  navigation  design,  (e)  allows  easy  access  to  the  visually  impaired  and  others 
who  use  a text  reader  and/or  speech  software,  and  (e)  can  contain  any  element  desired,  as  long  as  it  is  possible  and 
there  is  sufficient  time.  Disadvantages  are  that  (a)  this  method  is  time-consuming  for  all,  including  the  professor;  (b) 
if  the  professor  is  not  a committed  member  of  the  team,  expectations  may  be  unreasonable;  (c)  if  one  team  member 
leaves  or  graduates,  it  may  slow  down  production,  as  the  replacement  will  need  time  to  become  a member  of  the 
team. 

The  software  tool  approach  to  course  development 

For  those  who  choose  the  software  solution,  the  Instruction  Support  Education  area  currently  offers  a choice 
of  three  tools.  We  do  not  believe  at  this  time  that  it  is  necessary  or  wise  to  offer  only  one,  as  any  software  tool  has  its 
advantages  and  disadvantages.  While  the  standard  packages  have  improved  greatly,  none  really  offers  all  of  the 
features  one  would  desire.  An  intensive  look  at  the  tools  and  packages  being  offered  reveals  that  there  are  a number 
of  considerations  when  deciding  on  which  tool  to  use  or  to  test.  We  have  compiled  a list  of  questions  that  faculty  and 
staff  should  ask  themselves  before  making  this  decision. 
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1.  First,  was  the  tool  developed  originally  for  the  corporate  world  or  the  academic  world?  If  for  the  corporate  world,  will  its 
use  conflict  with  sound  instructional  theory?  If  you  are  not  sure,  ask  someone  in  the  College  of  Education  who  is  familiar 
with  computer-based  instruction  or  web-based  instruction. 

2.  Is  the  interface  intuitive  and  professional  for  both  students  and  professors?  Can  you,  as  instructor,  use  your  own  graphics? 
If  not,  there  is  a danger  that  all  courses  will  look  the  same  from  year  to  year,  so  that  students  forget  where  they  learned 
important  content.  If  possible,  ask  a graphics  design  professional  to  look  at  the  interface  and  give  an  opinion  on  design 
and  usability. 

3.  Which  interactive  features  does  it  offer;  which  do  you  see  a need  for?  You  may  want  a forum,  online  quizzes,  a chat  room. 
Sometimes  the  programming  that  is  necessary  to  build  in  interactivity  slows  the  speed  with  which  your  users  can  access 
your  course.  If  you  elect  to  incorporate  interactivity,  do  these  features  interfere  with  speed?  You  may  want  to  ask  an  expert 
from  Engineering  or  Computer  Science. 

4.  Does  the  package  provide  for  data  collection  and  user  tracking?  Depending  on  the  software  tool,  you  may  be  able  to  give 
quizzes,  add  quiz  scores  to  a grade  book  automatically,  check  on  which  pages  have  been  accessed  by  which  students,  and 
keep  logs  of  chat  room  activity.  Again,  sometimes  the  programming  that  is  necessary  to  build  in  such  data  collection  slows 
the  speed  with  which  your  users  can  access  your  course.  You  may  want  to  check  with  an  expert  from  Engineering  or 
Computer  Science  to  see  if  these  features  unacceptably  interfere  with  speed  of  access. 

5.  What  is  the  pricing?  How  does  this  compare  to  other  packages? 

6.  Does  it  offer  accessibility  for  the  visually  impaired?  Graphical  interfaces  can  present  difficulties  for  the  visually  impaired 
(Coombs,  1995).  If  a disabled  student  is  in  your  course,  you  will  need  to  offer  all  information  in  a manner  that  the  student 
can  access.  It  is  relatively  easy  to  write  web  pages  so  that  they  can  be  read  either  by  a text  reader  or  built-in  speech,  but  not 
all  web  course  tools  offer  this  ability. 

7.  How  many  users  and  classes  will  it  support?  Software  may  allow  many  courses  to  be  created,  but  break  down  in  actual  use 
if  a relatively  small  number  of  students  attempts  to  login  at  once.  You  may  have  to  test  the  software  on  the  server  you  will 
be  using  in  order  to  know  how  many  simultaneous  logins  the  software  can  support. 

8.  Finally,  does  it  work?  If  it  is  buggy,  will  students  believe  that  they  are  not  being  treated  fairly?  Again,  ask  an  expert  from 
Computer  Science  to  test  with  you. 

To  summarize,  there  are  both  advantages  and  disadvantages  to  the  use  of  web  course  creation  software 
tools.  Advantages  are  that  (a)  they  are  relatively  easy  to  use,  as  designer  does  not  need  to  learn  HTML  but  only  the 
program  that  creates  the  HTML;  (b)  they  provide  interactivity  with  no  extra  effort  on  the  part  of  the  instructor;  (c) 
they  provide  data  collection  to  varying  degrees;  and  (d)  built-in  password  protection  allows  a sense  of  protected 
community.  Disadvantages  to  take  into  account  are  that  (a)  courses  developed  with  such  a program  tend  to  look 
alike,  since  they  use  an  interface  created  by  the  program;  (b)  courses  may  look  old-fashioned  quickly,  a problem  that 
could  be  alleviated  if  the  creators  of  the  software  used  new  templates  each  year;  (c)  interactivity  and  data  collection 
are  done  via  programming  that  may  slow  down  the  speed  with  which  the  pages  are  accessed;  (d)  if  there  are  software 
bugs,  they  may  not  be  easily  remedied. 

Discussion 

During  two  and  one  half  years  of  development  the  road  has  not  always  been  smooth.  We  have  had  to  adjust 
and  change  policies  innumerable  times,  and  from  a bird’s-eye  view  of  that  road  we  can  now  offer  a number  of 
“lessons  learned”  that  we  can  share  with  others.  To  begin  with  many  difficulties  lie  in  the  fact  that  there  is  too  much 
information  to  manage  (Stoll,  1995).  It  has  been  suggested  (Applehans  and  Laugero,  199  ) that  it  is  beneficial  to 
think  in  terms  of  knowledge  management  rather  than  information  management,  where  knowledge  refers  to 
documented,  interpreted  information  that,  once  interpreted,  becomes  useful.  Knowledge  management  then  refers  to 
defining  job  functions,  identifying  the  knowledge  needed  for  each  job,  and  ensuring  that  workers  receive  that 
knowledge.  Applying  this  idea  to  the  Lab,  we  see  that  along  with  programmers  and  creative  artists,  a third  important 
job  function  is  the  administrative/clerical  area.  People  who  enjoy  detail-oriented  organizational  tasks,  and  can  relate 
to  these  tasks  in  a creative  and  technological  environment,  are  as  necessary  to  functioning  as  a lead  programmer.  We 
have  learned  that  unless  we  make  the  three  roles  equal  in  status  and  responsibility,  development  will  not  proceed 
smoothly. 

Documentation  and  data  collection  are  a high  priority  when  developing  in  an  environment  in  which  team 
members  may  be  collaborating  electronically,  when  new  paradigms  are  being  tested,  when  pedagogies  are  being 
extended  to  other  media,  and  when  students  are  relying  on  our  expertise  to  help  faculty  members  offer  web  courses. 
While  all  this  adds  information  to  manage,  it  is  important  for  predicting  where  we  are  going  and  identifying  issues 
that  need  to  be  addressed.  Giving  high  priority  to  the  administrative/clerical  side  of  the  Lab  has  helped,  as  has  an 
increase  in  supervisory  staff.  In  1996,  the  Lab  had  only  one  supervisor.  This  rapidly  proved  to  be  a difficult  situation 
as  one  person  cannot  efficiently  ensure  that  all  policies  and  procedures  are  followed,  guide  the  development  of  all 
projects,  supervise  all  employees,  be  a bridge  to  production  areas,  and  successfully  conduct  documentation.  The 
addition  of  an  assistant  supervisor  as  well  as  a graduate  student  who  can  act  as  assistant  supervisor  has  helped 
immensely.  However,  ensuring  good  communication  and  free  information  sharing  among  supervisors  becomes  of 
paramount  importance  so  that  policies  and  procedures  remain  standard. 

The  Instruction  Support  lab  remains  a drop-in  lab  for  faculty  requiring  immediate  help  as  well  as  being  a 
development  lab  for  those  creating  web  courses  and  other  projects.  This  has  its  advantages  and  disadvantages:  we 
gather  new  ideas  from  all  parts  of  the  University  by  thus  keeping  our  doors  open.  Moreover,  we  have  a better  idea  of 
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the  level  of  familiarity  with  technology  among  faculty  than  we  would  were  we  solely  a production  environment.  On 
the  other  hand,  it  can  be  difficult  to  meet  a deadline  for  a meeting  with  a faculty  member  when  team  members  must 
take  the  time  to  help  drop-in  clients  as  well.  Having  other  support  areas  within  Instruction  Support  that  we  can  call 
upon  if  necessary  is  one  way  in  which  we  have  been  able  to  deal  with  this  problem.  A quiet  space  where  students 
could  retreat  to  work  uninterruptedly  would  be  helpful  also.  Further,  while  keeping  our  door  open  to  drop-in  faculty 
members  helps  us  judge  the  level  of  support  needed,  it  does  not  help  with  those  faculty  members  who  are  afraid  to 
enter  the  doors.  Having  the  Lab  supervisor  go  directly  to  a College  to  support  technology  integration  there  has  been 
helpful  in  bringing  to  light  the  technology  problems  of  those  who  are  just  beginning  to  use  it. 

In  a University  setting  such  as  ours  it  is  common  for  some  staff  to  be  student  workers.  We  found  that  while 
the  majority  are  committed  and  dedicated,  a few  cannot  handle  the  freedom  of  a professional  development 
environment  in  which  they  must  make  many  of  their  own  decisions  and  manage  their  own  time.  We  have  found  it 
useful  to  place  the  student  under  the  direct  supervision  of  a student  who  is  doing  well;  the  second  student  then 
reports  to  the  supervisor  on  progress.  In  this  way  the  student  who  is  having  trouble  can  learn  from  a student  who  has 
adjusted  to  the  professional  model.  Sometimes  a student  having  difficulty  with  time  management  will  fail  to 
accomplish  tasks;  other  students  will  must  learn  to  avoid  burning  out.  Similarly,  some  students  find  it  difficult  to 
adjust  to  the  idea  that  they  will  be  working  in  a team.  Here,  it  is  useful  to  seek  and  hire  students  who  understand  the 
value  of  having  experience  in  a multidisciplinary  team,  and  who  are  eager  to  learn  to  work  with  their  peers  from 
other  Colleges. 

No  discussion  of  lessons  learned  would  be  complete  without  a reminder  to  back  up  data.  The  early  status  of 
the  Lab  as  a developmental,  experimental  venue  worked  well  when  faculty  members  who  co-developed  were  among 
the  early  adopters  of  technology.  As  the  users  of  Lab  services  grew  in  numbers,  many  were  not  expert.  In  time  we 
realized  that  we  would  need  to  treat  any  project  developed  for  a course  as  “production”  rather  than  “developmental” 
since  not  all  faculty  would  understand  the  difference.  Any  course  element  being  developed  in  the  Lab  must  now  be 
moved  onto  a production  server  located  in  a secure  area  before  being  offered  to  students. 
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